Analog neuromorphic circuits for dot-product operation implementing resistive memories

ABSTRACT

An analog neuromorphic circuit is disclosed having resistive memories that provide a resistance to each corresponding input voltage signal. Input voltages are applied to the analog neuromorphic circuit. Each input voltage represents a vector value that is a non-binary value included in a vector that is incorporated into a dot-product operation with weighted matrix values included in a weighted matrix. A controller pairs each resistive memory with another resistive memory. The controller converts each pair of resistance values to a single non-binary value. Each single non-binary value is mapped to a weighted matrix value included in the weighted matrix that is incorporated into the dot-product operation with the vector values included in the vector. The controller generates dot-product operation values from the dot-product operation with the vector and the weighted matrix where each dot-product operation is a non-binary value.

RELATED APPLICATIONS

The present application is a continuation application of U.S.Nonprovisional application Ser. No. 16/506,145 filed on Jul. 19, 2019,which claims the benefit of U.S. Pat. No. 10,346,738 filed on Jan. 8,2019, which claims the benefit of U.S. Pat. No. 10,176,425 filed on Jul.14, 2017 which claims the benefit of and priority to U.S. ProvisionalPatent Application No. 62/362,379 filed on Jul. 14, 2016 and U.S.Provisional Patent Application 62/366,379 filed on Jul. 25, 2016, thedisclosures of which are both incorporated by reference in theirentirety.

FIELD OF THE INVENTION

This invention relates to neural networks, and more particularly, tosystems and methods for implementing resistive memories in an analogneuromorphic circuit.

BACKGROUND OF THE INVENTION

Traditional computing systems use conventional microprocessor technologyin that operations are performed in chronological order such that eachoperation is completed before the subsequent operation is initiated. Theoperations are not performed simultaneously. For example, an additionoperation is completed before the subsequent multiplication operation isinitiated. The chronological order of operation execution limits theperformance of conventional microprocessor technology. Conventionalmicroprocessor design is limited in how small the microprocessors can bedesigned, the amount of power that the microprocessors consume, as wellas the speed in which the microprocessors execute operations inchronological order. Thus, conventional microprocessor technology isproving insufficient in applications that require high computationpower, such as in image recognition.

It is becoming common wisdom to use conventional neuromorphic computingnetworks which are laid out in a similar fashion as the human brain.Hubs of computing power are designed to function as a neuron in thehuman brain where different neurons of computing power are coupled toother neurons of computing power. This coupling of neurons enables theneuromorphic computing network to execute multiple operationssimultaneously. Therefore, the neuromorphic computing network hasexponentially more computing power than traditional computing systems.

Conventional neuromorphic computing networks are implemented in largescale computer clusters which include computers that are physicallylarge in order to attain the computation power necessary to executeapplications such as image recognition. For example, applications ofthese large scale computer clusters include rows and rows of physicallylarge servers that may attain the computation power necessary to executeimage recognition when coupled together to form a conventionalneuromorphic computing network. Such large scale computer clusters notonly take up a significant amount of physical space but also requiresignificant amounts of power to operate.

The significant amount of physical space and power required to operateconventional neuromorphic computing networks severely limits the typesof applications for which conventional neuromorphic computing networksmay be implemented. For example, industries such as biomedical,military, robotics, and mobile devices are industries that cannotimplement conventional neuromorphic computing networks due to thesignificant space limitations in such industries as well as the powerlimitations. Therefore, an effective means to decrease the space and thepower required by conventional neuromorphic computing is needed.

SUMMARY OF THE INVENTION

The present invention provides an analog neuromorphic circuit thatimplements a plurality of resistive memories, a plurality of inputvoltages and a controller. A plurality of input voltages is applied tothe analog neuromorphic circuit. Each input voltage represents a vectorvalue that is a non-binary value included in a vector that isincorporated into a dot-product operation with a plurality of matrixvalues included in a matrix. Each resistive memory is configured toprovide a resistance value to each corresponding input voltage. Eachresistance value is a positive resistance value selected from a finiterange of resistance values. The controller is configured to pair eachresistive memory with another resistive memory so that each pair ofresistive memories includes a pair of resistance values. The controlleris also configured to convert each pair of resistance values from a pairof resistance values selected from the finite range of resistance valuesto a single non-binary value. Each single non-binary value is mapped toa matrix value included in the matrix that is incorporated into thedot-product operation with the vector values included in the vector. Thecontroller is also configured to generate a plurality of dot-productoperation values from the dot-product operation with the vector and thematrix where each dot-product operation value is a non-binary value.

The present invention also provides a method for adjusting resistancesof a plurality of resistive memories positioned in an analogneuromorphic circuit. The method starts with applying input voltages tothe analog neuromorphic circuit. Each input voltage represents a vectorvalue that is a non-binary value included in a vector that isincorporated into a dot-product operation with a plurality of matrixvalues included in a matrix. The method further includes providing aresistance value to each corresponding input voltage from acorresponding resistance memory included in a plurality of resistivememories. Each resistance value is a positive resistance value selectedfrom a finite range of resistance values. The method further includespairing each resistive memory with another resistive memory from theplurality of resistive memories so that each pair of resistive memoriesincludes a pair of resistance values. The method further includesconverting each pair of resistance values from a pair of resistancevalues selected from the finite range of resistance values to a singlenon-binary value. The method further includes mapping each singlenon-binary value to a matrix value included in the matrix that isincorporated into the dot-product operation with the vector valuesincluded in the vector. The method further includes generatingdot-product operation values from the dot-product operation with thevector and the matrix. Each dot-product operation value is a non-binaryvalue.

The present invention also provides an analog neuromorphic system thatimplements a plurality of resistive memories, a plurality of inputvoltages, and a controller. The input voltages are applied to the analogneuromorphic circuit. Each input voltage represents an image value thatis a non-binary value included in an image matrix that is converted intoa voltage vector. Each resistive memory is configured to provide aresistance value to each corresponding input voltage. Each resistancevalue is mapped to a corresponding kernel value that is a non-binaryvalue included in a kernel matrix. A controller is configured to converteach image value included in the image matrix into the correspondinginput voltage that is included in a voltage vector. The controller isalso configured to map each corresponding kernel value to acorresponding resistance value associated with a corresponding resistivememory. The controller is also configured to generate a plurality offiltered image values from a dot-product operation with the image matrixrepresented by the plurality of input voltages and the kernel matrixrepresented by each resistance value associated with each of theresistive memories where the filtered image values depict a filteredimage.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate embodiments of the invention and,together with a general description of the invention given above, andthe detailed description given below, serve to explain the invention.Additionally, the left most digit(s) of a reference number identifiesthe drawing in which the reference number first appears.

FIG. 1 is a schematic illustration of exemplary analog neuromorphicprocessing device that simultaneously executes several computingoperations in parallel in accordance with an embodiment of thedisclosure;

FIG. 2 is a schematic illustration of an exemplary analog neuromorphiccircuit that simultaneously executes several computing operations inparallel in accordance with an embodiment of the disclosure;

FIG. 3 is a schematic illustration of an exemplary neural networkconfiguration that the analog neuromorphic circuit of FIG. 1 may beimplemented and scaled in accordance with an embodiment of thedisclosure;

FIG. 4 is a schematic illustration of an exemplary an analogneuromorphic circuit that may be implemented to execute dot-productoperations in a similar manner as a conventional computing device inaccordance with an embodiment of the disclosure;

FIG. 5 is a schematic illustration of an exemplary output configurationas depicted with a column of an analog neuromorphic circuit that may beimplemented to convert the output voltage of the column generated fromthe execution of dot-product operations to a dot-product operation valuein accordance with an embodiment of the disclosure;

FIG. 6 is a schematic illustration of an op-amp configuration that maygenerate a pseudo sigmoid function in accordance with an embodiment ofthe disclosure;

FIG. 7A is a schematic illustration of a resistance adjusterconfiguration as depicted with a column of the analog neuromorphiccircuit that may be implemented to adjust the resistance values of theresistive memories in accordance with an embodiment of the disclosure;

FIG. 7B is a schematic illustration of a resistance adjuster that maydetermine whether each of the resistance values for each correspondingresistive memory have corresponding conductance values that are withinthe maximum conductance level σ_(max) and the minimum conductance levelσ_(min) in accordance with an embodiment of the disclosure;

FIG. 8 is a schematic illustration of a conventional ConvolutionalNeural Network (CNN) that may be executed by analog neuromorphicconfigurations;

FIG. 9 is a flowchart that outlines an operation that may be performedby the analog neuromorphic circuit in accordance with an embodiment ofthe disclosure;

FIG. 10 is a schematic illustration of an analog neuromorphic circuit inaccordance with an embodiment of the disclosure;

FIG. 11 is a schematic illustration of an analog neuromorphic circuit inaccordance with an embodiment of the disclosure;

FIG. 12 is a schematic illustration of an analog neuromorphic circuit inaccordance with an embodiment of the disclosure;

FIG. 13 is a schematic illustration of an analog neuromorphic circuitwhere the entire image may be applied as inputs without having to breakeach of the feature maps into sections to generate each of the reducedpixel maps in accordance with an embodiment of the disclosure;

FIG. 14 is a schematic illustration of an analog neuromorphic circuitwhere the entire reduced pixel map may be applied as inputs withouthaving to break each of the reduced pixel maps into sections to generateeach of the output feature maps in accordance with an embodiment of thedisclosure;

FIG. 15 is a schematic illustration of an analog neuromorphic circuitwhere the initial analog neuromorphic circuit is broken downhorizontally into smaller analog neuromorphic circuits in accordancewith an embodiment of the disclosure; and

FIG. 16 is a schematic illustration of an analog neuromorphic circuitwhere the initial analog neuromorphic circuit is broken down verticallyinto smaller analog neuromorphic circuits in accordance with anembodiment of the disclosure.

DETAILED DESCRIPTION

The following Detailed Description refers to accompanying drawings toillustrate exemplary embodiments consistent with the present disclosure.References in the Detailed Description to “one embodiment,” “anembodiment,” “an exemplary embodiment,” etc., indicate that theexemplary embodiment described can include a particular feature,structure, or characteristic, but every exemplary embodiment does notnecessarily include the particular feature, structure, orcharacteristic. Moreover, such phrases are not necessarily referring tothe same embodiment. Further, when a particular feature, structure, orcharacteristic is described in connection with an embodiment, it iswithin the knowledge of those skilled in the relevant art(s) to affectsuch feature, structure, or characteristic in connection with otherexemplary embodiments whether or not explicitly described.

The exemplary embodiments described herein are provided for illustrativepurposes, and are not limiting. Other embodiments are possible, andmodifications can be made to exemplary embodiments within the scope ofthe present disclosure. Therefore, the Detailed Description is not meantto limit the present disclosure. Rather, the scope of the presentdisclosure is defined only in accordance with the following claims andtheir equivalents.

Embodiments of the present invention may be implemented in hardware,firmware, software, or any combination thereof. Embodiments of thepresent invention may also be implemented as instructions stored on amachine-readable medium, which may be read and executed by one or moreprocessors. A machine-readable medium may include any mechanism forstoring or transmitting information in a form readable by a machine(e.g., a computing device). For example, a machine-readable medium mayinclude read only memory (ROM); random access memory (RAM); magneticdisk storage media; optical storage media; flash memory devices;electrical, optical, acoustical or other forms of propagated signals(e.g., carrier waves, infrared signals, digital signals, etc.), andothers. Further, firmware, software, routines, and/or instructions maybe described herein as performing certain actions. However, it should beappreciated that such descriptions are merely for convenience and thatsuch actions in fact result from computing devices, processors,controllers, or other devices executing the firmware, software,routines, instructions, etc.

For purposes of this discussion, each of the various componentsdiscussed may be considered a module, and the term “module” shall beunderstood to include at least one of software, firmware, and hardware(such as one or more circuit, microchip, or device, or any combinationthereof), and any combination thereof. In addition, it will beunderstood that each module may include one, or more than one, componentwithin an actual device, and each component that forms a part of thedescribed module may function either cooperatively or independently ofany other component forming a part of the module. Conversely, multiplemodules described herein may represent a single component within anactual device. Further, components within a module may be in a singledevice or distributed among multiple devices in a wired or wirelessmanner.

The following Detailed Description of the exemplary embodiments will sofully reveal the general nature of the present disclosure that otherscan, by applying knowledge of those skilled in the relevant art(s),readily modify and/or adapt for various applications such exemplaryembodiments, without undue experimentation, without departing from thescope of the present disclosure. Therefore, such adaptations andmodifications are intended to be within the meaning and plurality ofequivalents of the exemplary embodiments based upon the teaching andguidance presented herein. It is to be understood that the phraseologyor terminology herein is for the purpose of description and notlimitation, such that the terminology or phraseology of the presentspecification is to be interpreted by those skilled in relevant art(s)in light of the teachings herein.

The present invention creates an analog neuromorphic computing networkby implementing resistive memories. A resistive memory is anon-volatile, variable resistor that may not only change the resistancelevel but may also maintain the resistance level after power to theresistive memory has been terminated so that the resistive memory actsas memory. The resistive memory may also have resistances that arepositive and negative. In an embodiment, an equivalent of a negativeresistance generated by a resistive memory may be generated byimplementing negative weight values with a pair of resistive memories.The output of one of the resistive memories from the pair may then beinverted by an inverting circuit. Such characteristics of the resistivememory enables neuromorphic computing to be shrunk down fromimplementing large computers to a circuit that can be fabricated onto achip while requiring minimal power due to the analog characteristics ofthe resistive memory.

The resistive memories may be positioned in a crossbar configuration inthat each resistive memory is positioned at an intersection of aplurality of horizontal wires and a plurality of vertical wires forminga wire grid. An input voltage may be applied to each horizontal wire.Each resistive memory may apply a resistance to each input voltage sothat each input voltage is multiplied by each resistance. Thepositioning of each resistive memory at each intersection of the wiregrid enables the multiplying of each input voltage by the resistance ofeach resistive memory to be done in parallel. The multiplication inparallel enables multiple multiplication operations to be executedsimultaneously. Each current relative to each horizontal wire may thenbe added to generate an accumulative current that is conducted by eachvertical wire. The addition of each current to generate the accumulativecurrents is also done in parallel due to the positioning of theresistive memories at each intersection of the wire grid. The additionin parallel also enables multiple addition operations to be executedsimultaneously. The simultaneous execution of addition andmultiplication operations in an analog circuit generates significantlymore computation power than conventional microprocessors whileimplementing significantly less power than conventional microprocessors.

The terms “horizontal” and “vertical” are used herein for ease ofdiscussion to refer to one example of the invention. It should beunderstood however that such orientation is not required, nor is aperpendicular intersection required. It is sufficient that a pluralityof parallel wires intersects a pair of parallel wires to form a crossbaror grid pattern having two wires for adding current and two or morewires for inputting voltages, with a resistive memory positioned at eachintersection for multiplication. The intersections may occur at rightangles (orthogonal crossing lines) or non-right angles. It may beunderstood, however, that the orthogonal arrangement provides thesimplest means for scaling the circuit to include additional neuronsand/or layers of neurons. Further, it may be understood that anorientation having horizontal rows and/or vertical columns is alsosimpler for scaling purposes and is a matter of the point of reference,and should not be considered limiting. Thus, any grid configurationorientation is contemplated.

Referring to FIG. 1, an analog neuromorphic processing device 100simultaneously executes several computing operations in parallel. Theanalog neuromorphic processing device 100 includes a plurality of inputvoltages 140(a-n) that are applied to a plurality of respective inputsof the analog neuromorphic processing device 100 and the analogneuromorphic processing device 100 then generates a plurality of outputsignals 180(a-n).

The analog neuromorphic processing device 100 may include a plurality ofresistive memories (not shown) that have variable resistancecharacteristics that may be exercised not only with low levels of powerbut may also exercise those variable resistance characteristics afterpower applied to the resistive memories has been terminated. Thevariable resistance characteristics of the resistive memories enable theresistive memories to act as memory while maintaining significantly lowpower requirements compared to conventional microprocessors. Theresistive memories are also of nano-scale sizes that enable asignificant amount of resistive memories to be configured within theanalog neuromorphic processing device 100 while still maintainingsignificantly low power level requirements. The variable resistancecapabilities of the resistive memories coupled with the nano-scale sizeof the resistive memories enable the resistive memories to be configuredso that the analog neuromorphic processing device 100 has significantcomputational efficiency while maintaining the size of the analogneuromorphic processing device 100 to a chip that may easily bepositioned on a circuit board.

For example, the resistive memories may include but are not limited tomemristors that are nano-scale variable resistance devices with asignificantly large variable resistance range. The physics of theresistive memories, such as memristors, require significantly low powerand occupy little space so that the resistive memories may be configuredin the analog neuromorphic processing device 100 to generate significantcomputational efficiency from a small chip.

The plurality of input voltages 140(a-n), where n is an integer greaterthan or equal to one, may be applied to corresponding inputs of theanalog neuromorphic processing device 100 to exercise the variableresistance characteristics of the resistive memories. The input voltages140(a-n) may be applied at a voltage level and for a time period that issufficient to exercise the variable resistance characteristics of theresistive memories. The input voltages 140(a-n) may vary and/or besubstantially similar depending on the types of variable resistancecharacteristics that are to be exercised by each of the resistivememories.

The resistive memories may be arranged in the analog neuromorphicprocessing device 100 such that the resistive memories maysimultaneously execute multiple addition and multiplication operationsin parallel in response to the input voltages 140(a-n) being applied tothe inputs of the analog neuromorphic processing device 100. Thevariable resistance characteristics of the resistive memories as well astheir nano-scale size enables a significant amount of resistive memoriesto be arranged so that the input voltages 140(a-n) trigger responses inthe resistive memories that are then propagated throughout the analogneuromorphic processing device 100 that results in simultaneousmultiplication and addition operations that are executed in parallel.

The simultaneous multiplication and addition operations executed inparallel exponentially increase the efficiency of analog neuromorphicprocessing device 100 while limiting the power required to obtain suchcomputation capabilities to the input voltages 140(a-n). The resistivememories are passive devices so that the simultaneous multiplication andaddition operations executed in parallel are performed in the analogdomain, which also exponentially decreases the required power. Forexample, the analog neuromorphic processing device 100 may havesignificantly more computational efficiency than traditionalmicroprocessor devices, and may be smaller than traditionalmicroprocessor chips while reducing power in a range from 1,000 times to1,000,000 times that of traditional microprocessors.

The resistive memories may also be arranged such that the simultaneousexecution of the multiplication and addition operations in parallel maybe configured as a single computation hub that constitutes a singleneuron in a neural network. The variable resistance characteristics andthe nano-scale size of the resistive memories further enable thearrangement of resistive memories to be scaled with other arrangementsof resistive memories so that the single neuron may be scaled into aneural network including multiple neurons. The scaling of a singleneuron into multiple neurons exponentially further increases thecomputational efficiency of the resulting neural network. In addition,the multiple neurons may be scaled into several layers of neurons thatfurther exponentially increases the computational efficiency of theneural network. The scaling of the resistive memories into additionalneurons may be done within the analog neuromorphic processing device 100such as within a single chip. However, the analog neuromorphicprocessing device 100 may also be scaled with other analog neuromorphiccircuits contained in other chips to exponentially increase thecomputational efficiency of the resulting neural network.

As a result, the analog neuromorphic processing device 100 may beconfigured into a neural network that has the capability of executingapplications with significant computational efficiency, such as imagerecognition. For example, the output signals 180(a-n), where n is aninteger greater than or equal to one, may generate signals thatcorrectly identify an image. The analog neuromorphic processing device100 may also have the learning capability as will be discussed infurther detail below so that analog neuromorphic circuits maysuccessfully execute learning neural network algorithms.

The analog neuromorphic processing device 100 implemented as a singleneuron and/or multiple neurons in a neural network and/or configuredwith other similar analog neuromorphic processing devices 100 may havesignificant advantages in traditional computing platforms that requiresignificant computational efficiency with limited power resources andspace resources. For example, such traditional computing platforms mayinclude but are not limited to Fast Fourier Transform (FFT)applications, Joint Photographic Experts Group (JPEG) imageapplications, and/or recognition, mining, and synthesis (RMS)applications. The implementation of low power neural networks that havea limited physical footprint may also enable this type of computationalefficiency to be utilized in many systems that have traditionally notbeen able to experience such computational efficiency due to the highpower consumption and large physical footprint of conventional computingsystems. Such systems may include but are not limited to military andcivilian applications in security (image recognition), robotics(navigation and environment recognition), and/or medical applications(artificial limbs and portable electronics).

The layering of the analog neuromorphic processing device 100 with othersimilar analog neuromorphic circuits may enable complex computations tobe executed. The compactness of the resistive memory configurationsenables fabrication of chips with a high synaptic density in that eachchip may have an increased amount of neurons that are fitted onto thechip. The passive characteristics of the resistive memories eliminatethe need for software code which increases the security of the analogneuromorphic processing device 100.

Referring to FIG. 2, an analog neuromorphic circuit 200 simultaneouslyexecutes several computing operations in parallel. The analogneuromorphic circuit 200 includes a plurality of resistive memories210(a-n) where n is an integer equal to or greater than four, aplurality of horizontal wires 220(a-n) where n is an integer equal to orgreater than two, a pair of vertical wires 230(a-b), a plurality ofinput voltages 240(a-n) where n is an integer equal to or greater thantwo, a pair of bias voltage connections 250(a-b), a first and secondinput of a comparator 260(a-b), a comparator 270, an output of thecomparator 280, a pair of weights 290(a-b), and a combined weight 295.The analog neuromorphic circuit 200 shares many similar features withthe analog neuromorphic processing device 100; therefore, only thedifferences between the analog neuromorphic circuit 200 and the analogneuromorphic processing device 100 are to be discussed in furtherdetail.

The analog neuromorphic circuit 200 may be representative of a singleneuron of a neural network. The analog neuromorphic circuit 200 has thecapability to be scaled to interact with several other analogneuromorphic circuits so that multiple neurons may be implemented in theneural network as well as creating multiple layers of neurons in theneural network. Such a scaling capability to include not only multipleneurons but also multiple layers of neurons significantly magnifies thecomputational efficiency of the neural network, as will be discussed infurther detail below.

The resistive memories 210(a-n) may be laid out in a crossbarconfiguration that includes a high density wire grid. The crossbarconfiguration enables the resistive memories 210(a-n) to be tightlypacked together in the wire grid as will be discussed in further detailbelow. The tightly packed resistive memories 210(a-n) provides a highdensity of resistive memories 210(a-n) in a small surface area of a chipsuch that numerous analog neuromorphic circuits may be positioned in aneural network on a chip while occupying little space. The crossbarconfiguration also enables the resistive memories 210(a-n) to bepositioned so that the analog neuromorphic circuit 200 may executemultiple addition and multiplication operations in parallel in theanalog domain. The numerous neuromorphic circuits may then be positionedin the neural network so that the multiple addition and multiplicationoperations that are executed in parallel may be scaled significantly,thus exponentially increasing the computational efficiency. Theresistive memories 210(a-n) are passive devices so that the multipleaddition and multiplication operations executed in parallel are done inthe analog domain, which also exponentially decreases the requiredpower.

As a result, the analog neuromorphic circuits that are configured into aneural network have the capability of executing applications requiringsignificant computation power, such as image recognition. The analogneuromorphic circuits also have learning capability as will be discussedin further detail below so that the analog neuromorphic circuits maysuccessfully execute learning algorithms.

Referring to FIG. 3, in which like reference numerals are used to referto like parts, neural network configuration 300 that the analogneuromorphic circuit 200 may be implemented and scaled into is shown.The neural network configuration 300 shares many similar features withthe analog neuromorphic processing device 100 and the analogneuromorphic circuit 200; therefore, only the differences between theneural network configuration 200 and the analog neuromorphic processingdevice 100 and the analog neuromorphic circuit 200 are to be discussedin further detail.

The analog neuromorphic circuit 200 may be implemented into the neuralnetwork configuration 300. The analog neuromorphic circuit 200 mayconstitute a single neuron, such as neuron 310 a in the neural networkconfiguration 300. As shown in FIG. 3, the input voltage 240 a andrepresented by “A” is applied to the horizontal wire 220 a, the inputvoltage 240 b and represented by “B” is applied to the horizontal wire220 b, and the input voltage 240 n and represented by “C” is applied tothe horizontal wire 220 c. The combined weight 295 as shown in FIG. 2 asrepresentative of the combined weight for the input voltage 240 a isshown as in FIG. 3. Similar combined weights for the input voltage 240 band the input voltage 240 n may also be represented in FIG. 3 in asimilar fashion. The wire grid, the resistive memories 210(a-n), and thecomparator 270 are represented by the neuron 310 a. The output 280 ofthe analog neuromorphic circuit 200 is coupled to additional neurons 320a and 320 b.

The analog neuromorphic circuit 200 may then be scaled so that similarcircuits may be configured with the analog neuromorphic circuit 200 toconstitute additional neurons, such as neurons 310(b-n) where n is aninteger greater than or equal to two. Each of the other neurons 310(b-n)includes similar circuit configurations as the analog neuromorphiccircuit 200. However, the resistances of the resistive memoriesassociated with each of the other neurons 310(b-n) may differ from theanalog neuromorphic circuit 200 so that outputs that differ from theoutput 280 of the analog neuromorphic circuit 200 may be generated.

Rather than limiting the input voltages 240(a-n) to be applied to asingle neuron 310, the input voltages 240(a-n) may also be applied tomultiple other neurons 310(b-n) so that each of the additional neurons310(b-n) also generate outputs that differ from the output 280 generatedby the analog neuromorphic circuit 200. The generation of multipledifferent outputs from the different neurons 310(a-n) exponentiallyincreases the computational efficiency of the neural networkconfiguration 300. As noted above, the analog neuromorphic circuit 200represented by the neuron 310 a operates as a single logic function withthe type of logic function being adjustable. The addition of neurons310(b-n) provides additional logic functions that also have thecapability of their logic functions being adjustable so that thecomputational efficiency of the neural network configuration 300 issignificant.

In addition to having several different neurons 310(a-n), the analogneuromorphic circuit 200 may also be scaled to include additional layersof neurons, such as neurons 320(a-b). The scaling of additional layersof neurons also exponentially increases the computational efficiency ofthe neural network configuration 300 to the extent that the neuralnetwork configuration 300 can execute learning algorithms. For example,a neural network configuration with a significant number of inputvoltages, such as several hundred, that are applied to a significantnumber of neurons, such as several hundred, that have outputs that arethen applied to a significant number of layers of neurons, such ashundreds, may be able to execute learning algorithms. The repetitiveexecution of the learning algorithms by the extensive neural networkconfiguration may result in the neural network configuration eventuallyattaining automatic image recognition capabilities.

For example, the neural network configuration may eventually output ahigh voltage value of “F₁” representative of the binary signal “1” andoutput a low voltage value of “F₂” representative of the binary signal“0” when the neural network configuration recognizes an image of a dog.The neural network configuration may then output a low voltage value of“F₁” representative of the binary signal “0” and output a high voltagevalue of “F₂” representative of the binary signal “1” when the neuralnetwork configuration recognizes an image that is not a dog.

However, the neural network configuration 300 does not automaticallyoutput a binary signal “1” for “F₁” and a binary signal “0” for “F₂”when the neural network configuration 300 recognizes an image of a dog.The neural network configuration 300 may have to execute learningalgorithms in millions of iterations until the resistance values of eachmemristor included in the neural network configuration 300 is at a valueso that the neural network configuration 300 outputs a binary signal “1”for “F₁” and a binary signal “0” for “F₂” when the neural networkconfiguration 300 recognizes the image of a dog.

Referring to FIG. 4, in which like reference numerals are used to referto like parts, an analog neuromorphic circuit 400 is shown that may beimplemented to execute dot-product operations in a similar manner as aconventional computing device. The analog neuromorphic circuit 400includes a plurality of resistive memories 410(a-n), a plurality ofinput voltages 440(a-n), a plurality of complemented input voltages460(a-n), a plurality of conductance 490 a, a plurality of complementedconductance 490 b, a plurality of amplifiers 480(a-n), a plurality ofdot-product operation values 470(a-n), and a plurality of complementeddot-product operation values 450(a-n). The analog neuromorphic circuit400 shares many similar features with the analog neuromorphic processingdevice 100, the analog neuromorphic circuit 200, and the neural networkconfiguration 300; therefore, only the differences between the analogneuromorphic circuit 400 and the analog neuromorphic processing device100, the analog neuromorphic circuit 200, and the neural networkconfiguration 300 are to be discussed in further detail.

The analog neuromorphic circuit 400 may be implemented so thatdot-product operations may be executed in a similar manner as aconventional computing system would execute dot-product operations bututilizing significantly less power than a conventional computing systemand requiring significantly less space than a conventional computingsystem. For example, conventional computing systems execute dot-productoperations in applications such as neural applications, imagerecognition, image processing, digital signal processing, video games,graphics and so on. In executing the dot-product operations, theconventional computing systems execute a vector/matrix multiplicationoperation where the conventional computing system takes values in avector format and executes a multiplication operation with values in amatrix format.

However, conventional computing systems are able to execute dot-productoperations when the values included in the vector format as well as thevalues included in the matrix format are non-binary numbers, such asfloating point numbers, such that the outputs of the executeddot-product operations are also non-binary numbers. Conventionalcomputing systems are not limited to simple values such as positiveinteger values. Rather, conventional computing systems are able toexecute dot-product operations with any floating point number whetherthose floating point numbers be positive or negative. For example, asimple electronic calculator is capable of multiplying a negativefloating point number of “−2.35965” with a positive floating pointnumber of “7.525” to generate a negative floating point number of“−17.5636625”.

As noted above, the analog neuromorphic circuit 400 may execute dotproduct operations with regards to non-binary numbers in a similarmanner as conventional computing devices without consuming significantamounts of power and/or incorporating significant amounts of space asconventional computing devices. In doing so, each value included in thevector that is requested to take part in the dot-product operation withthe matrix is converted to a voltage and is applied as an input voltage440(a-n) to each horizontal wire 220(a-n) as discussed in detail above.For example, the vector in Equation 1,

$\begin{matrix}\begin{matrix}2.35 \\{- 5.86} \\2.4\end{matrix} & (1)\end{matrix}$

may be applied to the analog neuromorphic circuit 400 where “2.35” isapplied as a voltage of 2.35V as input voltage 440 a to horizontal wire220 a, “−5.86” is applied as a voltage of −5.86V as input voltage 440 bto horizontal wire 220 b and “2.4” is applied as a voltage of 2.4V asinput voltage 440 n to horizontal wire 220 c.

Each value included in the matrix that is requested to take part in thedot-product operation with the vector may then be mapped to a resistancevalue associated with a resistive memory 410(a-n). For example, thematrix in Equation 2,

$\begin{matrix}{W = \begin{matrix}5.76 & {- 8.92} & 26.77 \\{- 100.25} & 2.59 & {- 1.23} \\56.89 & {- 9.25} & 17.88\end{matrix}} & (2)\end{matrix}$

may be mapped to the analog neuromorphic circuit 400 where “5.76” ismapped to the resistance value associated with resistive memory 410 a,“−8.92” is mapped to the resistance value associated with resistivememory 410 b, “26.77” is mapped to the resistance value associated withresistive memory 410 c, “−100.25” is mapped to the resistance valueassociated with resistive memory 410 d, “2.59” is mapped to theresistance value associated with resistive memory 410 e, “−1.23” ismapped to the resistance value associated with resistive memory 410 f,“56.89” is mapped to the resistance value associated with resistivememory 410 g, “−9.25” is mapped to the resistance value associated withresistive memory 410 h, and “17.88” is mapped to the resistance valueassociated with resistive memory 410 i.

As noted above, each resistance value associated with each resistivememory 440(a-n) may be adjusted within a range of resistance values.However, the range of resistance values that each resistive memory440(a-n) may be adjusted to is limited to positive values within afinite range of resistance values. For example, the resistance value ofeach resistive memory 440(a-n) may be adjusted to positive values withina range of 10,000 to 100,000. The limitation of resistance values foreach resistive memory 440(a-n) to positive values within a finite rangeof resistance values prevents a matrix that includes values such asnegative floating point numbers from being directly mapped onto theresistive memories 440(a-n). For example, the matrix value of “−8.92” inthe matrix shown in Equation 2 may not be directly mapped to resistivememory 410 b due to the possible resistance values of resistive memory410 b being unable to accommodate a negative value as well the floatingpoint number of “−8.92” being outside the range of the potentialresistance values of the resistive memory 410 b.

However, the controller 405 may adjust the resistance values of each ofthe resistive memories 410(a-n) such that the resistance values have theflexibility to accommodate non-binary values, such as positive and/ornegative floating point numbers, that are represented by the matrixvalues included in the matrix that is engaged in a dot-product operationwith the specified vector. For example, the controller 405 may adjustthe resistance values of the resistive memories 410(a-n) so that thematrix values included in the example matrix of Equation 2 may be mappedonto the resistive memories 410(a-n). In doing so, the analogneuromorphic circuit 400 may execute dot-product operations that involvepositive and/or negative floating point numbers included in the vectorand/or matrix and is also able to accurately generate the dot-productoperation values resulting from the dot-product operation of the vectorand matrix such that the generated dot-product operation values alsorepresent positive and/or negative floating point numbers. Thus, theanalog neuromorphic circuit 400 may execute dot product operations in asimilar manner as the conventional computing device in accommodatingpositive and/or negative floating point numbers but may do so withsignificantly less power and occupying significantly less space than theconventional computing device.

The controller 405 may adjust the resistance values of each of theresistive memories 410(a-n) to accommodate the non-binary valuesincluded in the matrix by pairing each resistive memory 410(a-n) withanother resistive memory so that each pair of resistive memoriesincludes a pair of resistance values. The controller 405 may thenconvert each pair of resistance values from a pair of positiveresistance values selected from the finite range of resistance values toa single non-binary value that is representative of a matrix valueincluded in the example matrix.

As noted above, each resistance value associated with each resistivememory 410(a-n) may be adjusted to a positive value within a finiterange of positive values. The controller 405 may then pair eachresistive memory and select a resistance value within the finite rangeof resistance values for each of the pair of resistive memories. Thecontroller 405 may then generate a relationship between the pair ofresistive memories such that a relationship between the resistance valueof the first resistive memory and the resistance value of the secondresistive memory represents a non-binary value, such as a positive ornegative floating point number. For example, the controller 405 may pairthe resistive memory 410 a and the resistive memory 410 d and select aresistance value of 1000 for resistive memory 410 a and a resistancevalue of 900 for 410 d. The controller 405 may then generate arelationship between the resistive memory 410 a and resistive memory 410d such that the relationship between the resistance value of 1000 andthe resistance value of 900 is substantially equivalent to the matrixvalue of “−8.92” in the example matrix of Equation 2.

In an embodiment, the controller 405 may map each positive matrix valueincluded in the matrix into a first relationship with the correspondingpair of resistive memories so that the first relationship converts thecorresponding pair of positive resistance values selected from thefinite range of resistance values to a positive single non-binary valuethat represents the positive matrix value. In such an embodiment, thematrix W may be transformed into a first matrix W⁺. Matrix W⁺ containspositive non-zero elements in each position where W_(ij) is greater than0 and zeroes are in all other positions. Therefore, W_(ij) ⁺=W_(ij) whenW_(ij) is greater than 0 and W_(ij) ⁺=0 when W_(ij) is less than orequal to 0. For example, the positive non-zero values in matrix W isincorporated into the matrix W⁺ in Equation 3,

$\begin{matrix}{W^{+} = \begin{matrix}5.76 & 0 & 26.277 \\0 & 2.59 & 0 \\56.89 & 0 & 17.88\end{matrix}} & (3)\end{matrix}$where each of the positive matrix values remain in the matrix and eachof the negative matrix values are replaced with zeroes.

The controller 405 may then identify a minimum resistance value and amaximum resistance value that corresponds to a pair of resistivememories that is associated with each positive matrix value incorporatedinto the first matrix and may select resistance values for each of theresistive memories that are within the minimum resistance value and themaximum resistance value. The controller 405 may then convert theselected resistance values for each pair of resistive memories into asingle non-binary value that is representative of the correspondingpositive matrix value based on the first relationship.

For example, the controller 405 may pair a resistive memory from theresistive memories 410(a-n) that have a conductance applied to them fromthe conductivity signal 490 a, with a resistive memory from theresistive memories 410(a-n) that have a negative conductance applied tothem from the conductivity, such as conductance signal 490 b. In such anexample, resistive memory 410 c that has a conductance applied to itfrom the conductance signal 490 a may be paired with resistive memory410 f that has a conductance applied to it from the conductance signal490 b and thereby associate the resistance values of resistive memories410 c and 410 f with the positive matrix value of 5.76 in Equation 3.The controller 405 may then identify a minimum resistance value and amaximum resistance value for the resistance values of resistive memories410 c and 410 f and select resistance values for the resistive memories410 c and 410 f that are within the minimum and maximum resistance valuerange for the resistive memories 410 c and 410 f. The controller 405 maythen convert the selected resistance values for resistive memories 410 cand 410 f into a single non-binary value that represents thecorresponding positive matrix value of 5.76 based on the firstrelationship.

The controller 405 may also map each negative matrix value included inthe matrix into a second relationship with the corresponding pair ofresistive memories so that the second relationship converts thecorresponding pair of positive resistance values selected from thefinite range of resistance values to a negative single non-binary valuethat represents the negative matrix value. In such an embodiment, thematrix W may be transformed into a second matrix W⁻. Matrix W⁻ containspositive non-zero elements in each position where W_(ij) is less than 0and zeroes are in all other positions. Therefore, W_(ij) ⁻=the absolutevalue of W_(ij) when W_(ij) is less than 0 and W_(ij) ⁻ is equal to 0and W_(ij) is greater than or equal to 0. For example, the positivenon-zero values in matrix W is incorporated into the matrix W⁻ inEquation 4,

$\begin{matrix}{W^{-} = \begin{matrix}0 & 8.92 & 0 \\100.25 & 0 & 1.23 \\0 & 9.25 & 0\end{matrix}} & (4)\end{matrix}$where each of the negative matrix values remain in the matrix aspositive values and each of the positive matrix values are replaced withzeroes.

The controller 405 may then identify a minimum resistance value and amaximum resistance value that corresponds to a pair of resistivememories that is associated with each negative matrix value incorporatedinto the second matrix and select resistance values for each of theresistive memories that are within the minimum resistance value and themaximum resistance value. The controller 405 may then convert theselected resistance values for each pair of resistive memories into asingle non-binary value that is representative of the correspondingnegative matrix value based on the second relationship.

For example, the controller 405 may pair a resistive memory from theresistive memories 410(a-n) that have a conductance applied to them fromthe conductance signal 490 a, with a resistive memory from the resistivememories 410(a-n) that have a conductance applied to them from theconductance signal 490 b. In such an example, resistive memory 410 gthat has a conductance applied from the conductance signal 490 a may bepaired with resistive memory 410 h that has a conductance applied to itfrom the conductance signal 490 b and thereby associate the resistancevalues of resistive memories 410 g and 410 h with the negative matrixvalue of −8.92 in Equation 4. The controller 405 may then identify aminimum resistance value and a maximum resistance value that theresistance values of resistive memories 410 g and 410 h and selectresistance values for the resistive memories 410 g and 410 h that arewithin the minimum and maximum resistance value range for the resistivememories 410 g and 410 h. The controller 405 may then convert theselected resistance values for resistive memories 410 g and 410 h into asingle non-binary value that represents the corresponding negativematrix value of −8.92 based on the second relationship.

In an embodiment, the controller 405 may map each positive input voltageand each positive matrix value into a co-linear first relationship withthe corresponding pair of resistive memories so that the co-linear firstrelationship converts a corresponding pair of conductance valuesselected from a finite range of conductance values to the positivesingle non-binary value that represents the positive weighted matrixvalue.

As noted above, each of the vector values included in the vector areapplied as input voltages 440(a-n). For example, with regards to thevector in Equation 1, “2.35” is applied as a voltage of 2.35V as inputvoltage 440 a to horizontal wire 220 a, “−5.86” is applied as a voltageof −5.86V as input voltage 440 b to horizontal wire 220 b and “2.4” isapplied as a voltage of 2.4V as input voltage 440 n to horizontal wire220 c. Each complement of the vector value may also be applied to theanalog neuromorphic circuit 400 as complemented input voltages 460(a-n).For example, with regards to the vector in Equation 1, the complement“−2.35” is applied as a voltage of −2.35V as complemented input voltage460 a to horizontal wire 220 d, the complement “5.86” is applied as avoltage of 5.86 as complemented input voltage 460 b to horizontal wire220 e, and the complement “−2.4” is applied as a voltage of −2.4V ascomplemented input voltage 460 n to horizontal wire 220 f.

With both the input voltages 440(a-n) representing the vector values aswell as the complemented input voltages 460(a-n) representing thecomplemented vector values applied to the analog neuromorphic circuit400, the controller 405 may generate a co-linear first relationshipbetween the corresponding pair of resistive memories to each positivematrix value. In order to generate the co-linear relationship, thecontroller 405 may identify the finite range of conductance values foreach pair of resistive memories. The conductance values of the resistivememories 410(a-n) is the inverse of the resistance values of theresistive memories 410(a-n). The controller may then determine aconductance value for each of the resistive memories included in thepair corresponding to the positive matrix value.

In an embodiment, each value in the matrix may be reproduced bysubtracting the second matrix W⁻ the first matrix W⁺ such that inEquation 5,W=W ⁺ −W ⁻.  (5)The conductance values of the resistive memories 410(a-n) may then beimplemented to represent the values provided in the matrix W. Thus, thevalue included in the matrices W+ and W may be converted to a boundednumber within the conductance range for each corresponding pair ofresistive memories in the analog neuromorphic circuit 400. Thecontroller 405 may then linearly scale the selected conductance valuefor each of the resistive memories included in the pair so thoseselected values are between the minimum conductance level 6 min and themaximum conductance level σ_(max) where the minimum conductance level 6min and the maximum conductance level σ_(max) represent the linearlyscaled minimum and maximum conductance values, respectively, for thepair of resistive memories.

The controller 405 may then determine a conductance value for each ofthe resistive memories included in the pair of resistive memoriescorresponding to the positive matrix value based on the co-linearrelationship provided in Equation 6,

$\begin{matrix}{\sigma^{+} = {{\frac{( {{\sigma\max} - {\sigma\min}} )}{\max( {{absolute}\mspace{14mu}{value}\mspace{14mu}{of}\mspace{14mu} W} )}W^{+}} + {\sigma_{m\; i\; n}.}}} & (6)\end{matrix}$and Equation 7,

$\begin{matrix}{\sigma^{-} = {{\frac{( {{\sigma\max} - {\sigma\min}} )}{\max( {{absolute}\mspace{14mu}{value}\mspace{14mu}{of}\mspace{14mu} W} )}W^{-}} + {\sigma_{m\; i\; n}.}}} & (7)\end{matrix}$The adding of the minimum conductance level σ_(min) to each of thevalues ensures that each of the zeros in the first matrix W⁺ and thesecond matrix W⁻ may be at the minimum conductance level for the pair ofresistive memories. The original matrix w may then be recovered fromEquation 8,

$\begin{matrix}{W = {\frac{\max( {{absolute}\mspace{14mu}{value}\mspace{14mu}{of}\mspace{14mu} W} )}{( {{\sigma\max} - {\sigma\min}} )}{( {\sigma^{+} - \sigma^{-}} ).}}} & (8)\end{matrix}$

For example, the controller 405 may incorporate equations 6 and 7 sothat a positive matrix value, such as positive matrix value 26.77 in theexample matrix of Equation 2, may generate a conductance signal that iswithin the minimum conductance level σ_(min) and the maximum conductancelevel σ_(max) of the pair of resistive memories corresponding to thepositive matrix value of 26.77. Since 26.77 is a positive matrix value,the controller 405 may determine the conductance values of σ⁺ and σ⁻ forthe pair of resistance values by incorporating equations 6 and 7 in thefollowing manner:

$\begin{matrix}{{\sigma^{+} = {{\frac{( {{\sigma\max} - {\sigma\min}} )}{\max( {{absolute}\mspace{14mu}{value}\mspace{14mu}{of}\mspace{14mu} W} )}( {2{6.7}7} )} + \sigma_{m\; i\; n}}},{and}} & (9) \\{\sigma^{-} = {{\frac{( {{\sigma\;\max} - {\sigma\;\min}} )}{\max( {{absolute}\mspace{14mu}{value}\mspace{14mu}{of}\mspace{14mu} W} )}(0)} + \sigma_{m\; i\; n}}} & (10)\end{matrix}$where a conductance value of σ⁺ and a conductance value of σ⁻ may begenerated by the controller 405 for the pair of resistive memoriesrepresenting the positive matrix value 26.77. The controller 405 maythen convert the conductance values of σ⁺ and σ⁻ into a positive singlenon-binary value that represents the positive matrix value of 26.77.

In a similar manner, the controller 405 may map each negative matrixvalue into a co-linear second relationship with the corresponding pairof resistive memories so that the co-linear second relationship convertsa corresponding pair of conductance values selected from the finiterange of conductance values to the negative single non-binary value thatrepresents the negative weighted matrix value.

For example, the controller 405 may incorporate equations 6 and 7 sothat a negative matrix value, such as negative matrix value −100.25 inthe example matrix of Equation 2, may generate a conductance signal thatis within the minimum conductance level σ_(min) and the maximumconductance level σ_(max) of the pair of resistive memoriescorresponding to the negative matrix value of −100.25. Since −100.25 isa negative matrix value, the controller 405 may determine theconductance values of σ⁺ and σ⁻ for the pair of resistance values byincorporating equations 6 and 7 in the following manner:

$\begin{matrix}{{\sigma^{+} = {{\frac{( {{\sigma\;\max} - {\sigma\;\min}} )}{\max\;( {{absolute}\mspace{14mu}{value}\mspace{14mu}{of}\mspace{14mu} W} )}(0)} + \sigma_{m\; i\; n}}},{and}} & (11) \\{\sigma^{-} = {{\frac{( {{\sigma\;\max} - {\sigma\;\min}} )}{\max( {{absolute}\mspace{14mu}{value}\mspace{14mu}{of}\mspace{14mu} W} )}( {- 100.25} )} + \sigma_{m\; i\; n}}} & (12)\end{matrix}$where a conductance value of σ⁺ and a conductance value of σ⁻ may begenerated by the controller 405 for the pair of resistive memoriesrepresenting the positive matrix value −100.25. The controller 405 maythen convert the conductance values of σ⁺ and σ⁻ into a negative singlenon-binary value that represents the negative matrix value of −100.25.

After the controller 405 has determined the conductance values σ⁺ and σ⁻for each pair of resistive memories included in the analog neuromorphiccircuit 400, the controller may adjust the resistance values of eachpair of resistive memories the conductance values σ⁺ and σ⁻ with theconductance signal 490 a and the conductance signal 490 b. Theadjustment of the resistance values of each pair of resistive memoriesto the conductance values σ⁺ and σ⁻ configures the analog neuromorphiccircuit 400 such that the resistive memories 410(a-n) represent thenon-binary values included in the matrix, such as the positive ornegative floating point numbers in the example matrix in Equation 2.

The dot-product operation with a vector, such as the example vector inEquation 1, and a matrix, such as the example matrix in Equation 2, maythen be executed incorporating the analog neuromorphic circuit 400. Asnoted above, each of the values in the vector may be applied as inputvoltages 440(a-n) to each corresponding horizontal wire 220(a-n).Currents may then be generated as each input voltage 440(a-n) is appliedto each resistance value associated with each resistive memory 440(a-n)as adjusted by the controller 405. The currents may then propagatethrough the analog neuromorphic circuit 400 in a fashion where the dotproduct operation is executed with the dot-product operation values470(a-n) generated as output values of the dot product operation. Thisprocess is discussed in detail in U.S. Nonprovisional application Ser.No. 15/082,537 and is incorporated by reference in its entirety.

With the conversion of the resistance values for the resistive memories410(a-n) to represent to represent the non-binary values included in amatrix, the analog neuromorphic circuit 400 is able to execute dotproduct operations in a similar manner as conventional computing systemsbut with consuming significantly less power and occupying significantlyless space than conventional computing systems. The analog neuromorphiccircuit 400 is capable of executing dot product operations in numerousapplications such as but not limited to neural applications, imagerecognition, image processing, digital signal processing, video games,graphics and so on.

For example, the analog neuromorphic circuit 400 may be incorporatedinto image processing applications where the vector represents an imageand the matrix includes a set of weighted values that are to be appliedto the image to improve the quality of the image that is to bedisplayed. Through numerous iterations, the resistance values of theresistive memories 410(a-n) may be adjusted until the resistance valuesaccurately represent the weighted values included in the matrix and thedot-product operation values 470(a-n) generated by the analogneuromorphic circuit 400 depict a high-quality display of the image.

In another example, the analog neuromorphic circuit 400 may beincorporated into digital signal processing applications where a filtermay be applied to an image to improve and/or change the quality of theimage. In doing so, the analog neuromorphic circuit 400 may execute aconvolution operation where the filter as defined by a kernel matrix isdirectly applied to a matrix depicting the image and generate an imagewith improved and/or changed quality without having to execute theneural operations where the resistance values of the resistive memories410(a-n) are adjusted through numerous iterations.

In such an example, the image that the filter is to be applied to isdefined in example image matrix x_(ex) in Equation 13,

$\begin{matrix}{x_{ex} = { \begin{bmatrix}0 & {0.5} & 0.3 \\0.5 & {0.8} & 0.5 \\0 & {0.5} & 0\end{bmatrix}arrow\begin{pmatrix}0 \\0.5 \\0.3 \\0.5 \\0.8 \\0.5 \\0 \\0.5 \\0\end{pmatrix} ->{\overset{x_{ex}}{\begin{pmatrix}0 \\0.5 \\0.3 \\0.5 \\0.8 \\0.5 \\0 \\0.5 \\0\end{pmatrix}}{\overset{x_{ex}}{\begin{pmatrix}0 \\{- 0.5} \\{- 0.3} \\{- 0.5} \\{- 0.8} \\{- 0.5} \\0 \\{- 0.5} \\0\end{pmatrix}}.}}}} & (13)\end{matrix}$The image is a two-dimensional image as depicted by the image matrixx_(ex). However, as noted in detail above, the analog neuromorphiccircuit 400 requires that the input values be represented in a vectorformat so that the vector values may be applied to the analogneuromorphic circuit 400 as input voltages 440(a-n) and complementedinput voltages 460(a-n). Thus, the controller 405 may convert the imagematrix x_(ex) into a vector such that the vector values may then beapplied to the analog neuromorphic circuit 400 as the input voltages440(a-n) and the complemented input voltages 460(a-n). With regards toEquation 14, the controller 405 may convert the image matrix x_(ex) intothe vector values included in the vector x_(ex) that are applied as theinput voltages 440(a-n) and the vector values included in the vector−x_(ex) that are applied as the complemented input voltages 460(a-n).

The filter that is to be directly applied to the image may be defined inthe example kernel matrix k_(ex) in Equation 14,

$\begin{matrix}{k_{ex} = {\begin{bmatrix}0.1 & {- 0.2} & 0.3 \\{- 0.4} & 0.5 & {- 0.6} \\0.7 & {- 0.8} & 0.9\end{bmatrix}->{\begin{pmatrix}0.9 \\{- 0.6} \\0.3 \\{- 0.8} \\0.5 \\{- 0.2} \\0.7 \\{- 0.4} \\0.1\end{pmatrix}->{\overset{k_{ex}^{+}}{\begin{pmatrix}0.9 \\0 \\0.3 \\0 \\0.5 \\0 \\0.7 \\0 \\0.1\end{pmatrix}}->{\overset{k_{ex}^{-}}{\begin{pmatrix}0 \\0.6 \\0 \\0.8 \\0 \\0.2 \\0 \\0.4 \\0\end{pmatrix}}.}}}}} & (14)\end{matrix}$The controller 405 may then convert the kernel matrix k_(ex) into k_(ex)⁺ and k_(ex) ⁻ which are similar to W⁺ and W⁻ discussed above. Thecontroller 405 may then determine the minimum conductance level σ_(min)and the maximum conductance level σ_(max) for each pair of resistivememories so that the values included in k_(ex) ⁺ and k_(ex) ⁻ may beaccurately represented by the resistive memories 410(a-n) in the analogneuromorphic circuit 400 as discussed in detail above. The convolutionoperation of passing through the image x_(ex) through the filter of thekernel matrix k_(ex) may then be executed with the analog neuromorphiccircuit 400 rather than a convention computing system.

Referring to FIG. 5, in which like reference numerals are used to referto like parts, an output configuration 500 as depicted with a column ofan analog neuromorphic circuit 400 that may be implemented to convertthe output voltage of the column generated from the execution ofdot-product operations to a dot-product operation value is shown. Theoutput configuration 500 includes the plurality of resistive memories410(a-n), the plurality of input voltages 440(a-n), the plurality ofcomplemented input voltages 460(a-n), the conductance 490 a, theconductance 490 b, an output voltage value 510, a first op-ampconfiguration 520 and a second op-amp configuration 530. The outputconfiguration 500 shares many similar features with the analogneuromorphic processing device 100, the analog neuromorphic circuit 200,the neural network configuration 300; and the analog neuromorphiccircuit 400 therefore, only the differences between the outputconfiguration 500, and the analog neuromorphic circuit 400, the analogneuromorphic processing device 100, the analog neuromorphic circuit 200,and the neural network configuration 300 are to be discussed in furtherdetail.

After the dot product operation has been executed, each column of theanalog neuromorphic circuit 400 generates an output voltage signal 510.The output voltage signal 510 is generated from each input voltage440(a-n) being applied to each corresponding horizontal wire 220(a-n)and then generating a current from each of the resistive values for eachresistive memory 410(a-n) that is then propagated through the analogneuromorphic circuit 400 as discussed in detail in U.S. Nonprovisionalapplication Ser. No. 15/082,537. The output voltage signal 510 that isgenerated as an output of each column in the analog neuromorphic circuit400 represents the dot-product operation values generated from the dotproduct operation of the vector and the matrix by the analogneuromorphic circuit 400. However, the output voltage signal 510 is avoltage and is yet to be converted to a non-binary value, such as apositive or negative floating point number, that is substantiallyequivalent to the dot-product operation value.

Positioning a comparator configuration at the output of each column ofthe analog neuromorphic circuit 400 compares the output voltage signalof each column to a desired output voltage signal and then generates abinary signal in a “0” or a “1” based on the comparison. The generatedbinary signal “0” or “1” may then be incorporated into the applicationthat the analog neuromorphic circuit 400 is being implemented to executesuch as correctly identifying whether an image depicts a “3”. Theconversion of the output voltage signal to a binary signal of a “0” or a“1” with a comparator configuration requires that each output voltagesignal be classified as binary value of a “0” or a “1”. With regards toimage classification applications, the conversion of each output voltagesignal to a binary value of a “0” or “1” may accurately identify simplerimages such as an image of a “3” but may become less accurate with morecomplicated images such as correctly identifying an image as a dog.

However, as noted above, the analog neuromorphic circuit 400 generatessignificantly more complicated output voltage signals with the executionof the dot product operations that accommodate non-binary values, suchas positive or negative floating point numbers. As a result, the outputvoltage signal 510 is a voltage value that represents the dot-productoperation value of the analog neuromorphic circuit 400 where thedot-product operation value is a non-binary value, such as a positive ornegative floating point number.

Rather than classifying the output voltage signal 510 as a binary valueof a “0” or a “1”, the output configuration 500 may be incorporated intothe analog neuromorphic circuit 400 to simulate a non-linear smoothfunction configuration 600 in FIG. 6, such as a sigmoid function, ascompared to a linear function that represents a binary output of a “0”or “1”. The simulation of the non-linear smooth function 610, such asthe sigmoid function, enables the output configuration 500 to convertthe output voltage signal 510 to the non-binary values represented bythe dot-product operation value 470 a and the complemented dot-productoperation value 450 a. In doing so, the output configuration 500 mayenable the analog neuromorphic circuit 400 to execute more accuratelycomplicated applications such as identifying an image of a dog. As shownin FIG. 6, the non-linear smooth function 610 may be a continuous smoothfunction between the values of 0 and 1, which may provide the vehicle inwhich the output voltage signal 510 may be converted to the non-binaryvalues represented by the dot-product operation value 470 a and thecomplemented dot-product operation value 450 a. The output configuration500 may convert the output voltage signal to model the sigmoid function,the inverse tangent function, and/or any other type of non-linear smoothfunction that is apparent to those skilled in the art.

The output configuration 500 includes the first op-amp configuration 520and the second op-amp configuration 530 that may be positioned at theoutput of each column of the analog neuromorphic circuit 400 to bothscale the output voltage signal 510 to a value on the non-linear smoothfunction 610 between “0” and “1” and does so by incorporating a neuronfunction such as an activation function and/or a thresholding function.The first op-amp configuration 520 may generate a pseudo sigmoidfunction, such as pseudo sigmoid functions 620(a-c) as shown in FIG. 6.The first op-amp configuration 520 may be configured such that theop-amp linear configuration 520 incorporates a linear amplifier transferfunction bounded by upper and lower voltage rails that are similar tothe desired bounds of the non-linear smooth function 600. For example,the first op-amp configuration 520 may incorporate a linear amplifiertransfer function that is bounded by an upper voltage of 1.0V and alower voltage of 0.0V.

The first op-amp configuration voltage 540 a may then be adjusted suchthat the first op-amp configuration 520 generates the pseudo sigmoidfunctions 620(a-c). For example, the first op-amp configuration voltage540 a may be adjusted to 0.3V such that the first op-amp configuration520 generates the pseudo sigmoid function 620 c. The pseudo sigmoidfunctions 620(a-c) may not be pure non-linear smooth functions butrather linear amplifier transfer functions that are adequately similarto the non-linear smooth function 610. The first op-amp configurationvoltage 540 a may be optimally adjusted such that the first op-ampconfiguration 520 generates the pseudo sigmoid functions 620(a-c) suchthat the pseudo sigmoid functions 620(a-c) are adequately similar to thenon-linear smooth function 610 to convert the output voltage signal 510to the complemented dot-product operation value 450 a. The second op-ampconfiguration voltage 540 b may be adjusted such that the second op-ampconfiguration 530 converts the output voltage signal 510 to thedot-product operation value 470 a.

In an embodiment, the first op-amp configuration 520 generating thecomplemented dot-product operation value 450 a may act as a summingamplifier that also accounts for the slope and bias required of theapproximate sigmoid activation function. The complemented dot-productoperation value 450 a may then be fed into the second op-ampconfiguration 530 that may be a unity gain amplifier to obtain thedot-product operation value 470 a. Equation 15 depicts an exampleembodiment of the first op-amp configuration 520,y _(j) ⁻ =R _(g)(Σ_(i=1) ^(N)[x _(i)σ_(ij) ⁺ −x _(i)σ_(ij) ⁻]+x_(N+1)σ_(b))  (15)The voltage x_(X+1)=1 may be used to drive the bias value b of thesigmoid function. The resistance R_(g) may be the resistance of theprogrammable gain resistor M_(g) that converts output voltage signal 510to the actual dot-product operation value represented by the dot-productoperation value 470 a and the complemented dot-product operation value450 a. The value σ_(b) is the conductance of the memristor M_(N+1) andσ_(b)=b/R_(g).

In Equation 16,

$\begin{matrix}{R_{g} = {m\frac{\max(W)}{( {\sigma_{m\;{ax}} - \sigma_{m\; i\; n}} )}}} & (16)\end{matrix}$R_(g) may be set so that the summation of the conductance and voltagepairs is multiplied by the inverse of the scaling factor in equations 6and 8 as well as the slope of the activation function m. As a result,the resulting complemented dot-product operation value 450 a and thedot-product operation value 470 a may have a value substantially equalto that of the dot-product operation performed by a conventionalcomputing system after the activation function has been applied.

In Equation 17, the dot-product operation value 470 a may be determinedfrom the complemented dot-product operation value 450 a,

$\begin{matrix}{{y_{j}^{+} = {{{- \frac{R_{f}}{Rf}}y_{j}^{-}} = {- y_{j}^{-}}}}.} & (17)\end{matrix}$Thus, the actual dot-product operation values may be generated from thedot product operation executed by the analog neuromorphic circuit 400.If additional layers of analog neuromorphic circuits are present, thedot-product operation value 470 a may be provided to the additionalanalog neuromorphic circuit as σ⁺ and the complemented dot-productoperation value 450 a may be provided to the additional analogneuromorphic circuit 400 as σ⁻.

Referring to FIG. 7A, in which like reference numerals are used to referto like parts, a resistance adjuster configuration 700 is depicted witha column of an analog neuromorphic circuit 400 that may be implementedto adjust the resistance values of the resistive memories 410(a-n) withthe resistance adjuster 750 in FIG. 7B. The resistance adjusterconfiguration 700 and the resistance adjuster 750 include the pluralityof resistive memories 410(a-n), the plurality of input voltages440(a-n), the output voltage value 510, the first op-amp configuration520, the second op-amp configuration 530, a first comparator 710 a, asecond comparator 710 b, and resistance adjusters 720(a-c). Theresistance adjuster configuration 700 and the resistance adjuster 750share many similar features with the analog neuromorphic processingdevice 100, the analog neuromorphic circuit 200, the neural networkconfiguration 300, the analog neuromorphic circuit 400, and the outputconfiguration 500, therefore, only the differences between theresistance adjuster configuration 700 and the resistance adjuster 750and the output configuration 500, the analog neuromorphic circuit 400,the analog neuromorphic processing device 100, the analog neuromorphiccircuit 200, and the neural network configuration 300 are to bediscussed in further detail.

After the minimum conductance level σ_(min) and the maximum conductancelevel σ_(max) for each pair of resistive memories have been determined,the resistance adjuster 750 in FIG. 7B may determine whether theresistance values for each corresponding resistive memories havecorresponding conductance values that are within the maximum conductancelevel σ_(max) and the minimum conductance level σ_(min). The resistanceadjuster 750 may then adjust the resistance values for each of theresistive memories that have corresponding conductance values that areoutside of the maximum conductance level σ_(max) and the minimumconductance level σ_(min) such that the resistive values of thoseresistive memories have corresponding conductance values that are withinthe maximum conductance level σ_(max) and the minimum conductance level6 min. The adjusting of the resistance values of resistive memories by aresistance adjuster is discussed in detail in U.S. Nonprovisionalapplication Ser. No. 15/202,995, which is incorporated herein byreference in its entirety.

In order to determine the current resistance value of a correspondingresistive memory, each of the input voltages 440(a-n) may be set to0.0V. However, an input voltage may be applied to the horizontal row220(a-n) where the resistive memory is positioned that has the requestedcurrent resistance value. For example, the current resistance value ofthe resistive memory 410 b is requested. The resistive memory 410 b ispositioned on horizontal wire 220 b. Each of the input voltages 440 a,440 c, and 440 n may be set to 0.0V while a positive voltage applied asinput voltage 440 b to the horizontal wire 220 b. The output voltagevalue 510 is then based on the positive voltage applied as input voltage440 b to the resistive memory 410 b due to each of the remaining inputvoltages 440 a, 440 c and 440 n set to 0.0V, which fails to activate theresistance values of resistive memories 410 a, 410 c, and 410 n. Thus,the output voltage value 510 is isolated to the resistive memory 410 band represents the current resistance value of the resistive memory 410b.

After the output voltage value 510 is applied to the first op-ampconfiguration 520 a, the complemented dot-product operation value 450 ais generated based on the single resistance value associated withresistive memory 410 b. The complemented dot-product operation value 450a may then be applied to a first comparator 710 a and a secondcomparator 710 b. The maximum conductance voltage signal 750 a and theminimum conductance voltage signal 750 b may signify tolerance boundsaround a target resistance value that is to be programmed byincorporating the resistance adjuster 750. The resistance adjuster 750may program a resistance until the complemented dot-product operationvalue 450 a from the first op-amp configuration 520 falls between thetolerance bounds that signify a maximum and minimum programmableconductance value that may differ from the conductance 490 a and theconductance 490 b, respectively. The maximum and minimum programmableconductance values may be tailored to each specific device such that themaximum and minimum programmable conductance values may differ betweendevices. However, the maximum conductance level σ_(max) and the minimumconductance level σ_(min) may be universal values among differentdevices and signify the maximum programming range for each of thedifferent devices.

The complemented dot-product operation value 450 a may then be comparedto the maximum conductance voltage signal 750 a as the complementeddot-product operation value 450 a and the maximum conductance voltagesignal 750 a are applied to the first comparator 710 a. The complementeddot-product operation value 450 a may also be compared to the minimumconductance voltage signal 750 b as the complemented dot-productoperation value 450 a and the minimum conductance voltage signal 750 bare applied to the second comparator 710 b.

The resistance adjuster 720 a may then generate an increased resistancesignal 760 a when the complemented dot-product operation value 450 a isless than the minimum conductance voltage signal 750 a. The increasedresistance signal 760 a may then increase the resistance value of theresistive memory 410 b until the resistance value corresponds to aconductance value that is within the maximum tolerable conductance leveland the minimum tolerable conductance level. The resistance adjuster 720c may generate a no-change resistance signal 760 c when the complementeddot-product operation value 450 a is within the minimum conductancevoltage signal 750 a and the maximum conductance voltage signal 750 b inwhich the current resistance value of the resistive memory 410 b remainsunchanged. The resistance adjuster 720 b may generate a decreasedresistance signal 760 b when the complemented dot-product operationvalue 450 a is above the maximum conductance voltage signal 750 b. Thedecreased resistance signal 760 b may then decrease the resistance valueof the resistive memory 410 b until the resistance value corresponds toa conductance value that is within the maximum tolerable conductancelevel and the minimum tolerable conductance level. The resistance valuesof each of the other resistive memories 410(a-n) may be determined insimilar manner as resistive memory 410 b and each of the resistivememories may be adjusted in a similar manner as resistive memory 410 b.

The analog neuromorphic circuit 400 may be incorporated into analogneuromorphic configurations with other analog neuromorphic circuits toexecute popular existing neural network algorithms. The crossbarconfiguration of the analog neuromorphic circuit 400 combined with theprogrammability capabilities of the resistive memories 410(a-n) mayenable the development of highly efficient neural systems. In doing so,the crossbar configuration is capable of performing N×M convolutionoperations in parallel where N is equal to the number of input maps andM is equal to the number of output maps in a given layer of a CNNsystem. For example, the analog neuromorphic circuit 400 may beincorporated into analog neuromorphic configurations to execute popularneural network algorithms that include but are not limited to aMultilayer Perceptron (MLP), a Restricted Boltzmann Machine (RBM),and/or a CNN.

In executing each of the neural network algorithms, the weights of eachof the resistive memories 410(a-n) may be determined as described indetail above by the controller 405 and then each of the resistive valuesfor each of the resistive memories 410(a-n) may be then adjusted asdescribed in detail above. Referring to FIG. 8, a conventional CNN 800that may be executed by analog neuromorphic configurations that includethe analog neuromorphic circuit 400 is depicted. The conventional CNN isa type of feed-forward neural network in which the connectivity patternbetween the neurons of the conventional CNN attempt to replicate theorganization of the animal visual cortex whose individual neurons arearranged in such a way that the neurons respond to overlapping regionstiling the visual field. The discussion below provides in detail howanalog neuromorphic configurations that include the analog neuromorphiccircuit 400 may be incorporated to execute the conventional CNN 800.However, analog neuromorphic configurations that include the analogneuromorphic circuit 400 may be incorporated to execute any other neuralnetwork algorithm that is apparent to those skilled in the art.

The conventional CNN 800 includes two main parts that are the featureextractor 810 and the classifier 820. The feature extractor 810 mayinclude several layers with each layer of the neural network executingthe feature extractor 810 receiving an input from the immediate previouslayer. The feature extractor 810 includes the combination of twodifferent types layers that are the convolution layers 830(a-n), where nis an integer equal to or greater than one, and the subsampling layers840(a-n), where n is an integer equal to or greater than one. Theoutputs of the convolution layers 830(a-n) and the subsampling layers840(a-n) are organized into multiple two-dimensional planes known asfeature maps. The convolution layers 830(a-n) extract the features fromthe input images using convolution operations and the subsampling layers840(a-n) abstract the feature maps through an averaging filter.

As the features propagate through the conventional CNN 800, the size ofthe features is reduced in terms of pixels depending on the size of theconvolution kernels and the subsampling kernels applied to the featuresby each of the neural layers, respectively. However, the number offeature maps is also increased so the conventional CNN 800 may determinethe most suitable features of the input images for better classificationaccuracy. The outputs of the last layer of the conventional CNN 800 arethen input to a fully connected network that is the classifier 820.

FIG. 9 is a flowchart of exemplary operational steps of an analogneuromorphic configuration that includes analog neuromorphic circuitssimilar to analog neuromorphic circuit 400 according to an exemplaryembodiment of the present invention. For discussion purposes, thefollowing exemplary operation steps are focused on the execution of theconventional CNN 800 where images are incorporated as inputs to theanalog neuromorphic configuration. However, the present invention is notlimited to this operational description. Rather, it will be apparent topersons skilled in the relevant art(s) from the teaching herein thatother operational control flows are within the scope of the presentinvention. For example, other operational control flows that incorporateany image as inputs to the analog neuromorphic configuration may beimplemented as well as implementing other neural network algorithms thatdiffer from the conventional CNN 800. The following discussion describesthe steps in FIG. 9.

At step 910, the operational control flow 900 executes a convolutionlayer by generating feature maps in parallel from an initial inputimage. Initially, an image is applied to the analog neuromorphic circuit1000 depicted in FIG. 10. The analog neuromorphic circuit 1000 sharesmany similar features with the analog neuromorphic processing device100, the analog neuromorphic circuit 200, the neural networkconfiguration 300, the analog neuromorphic circuit 400, and the outputconfiguration 500; therefore, only the differences between the analogneuromorphic circuit 1000 and the output configuration 500, the analogneuromorphic circuit 400, the analog neuromorphic processing device 100,the analog neuromorphic circuit 200, and the neural networkconfiguration 300 are to be discussed in further detail.

The image applied to the analog neuromorphic circuit 1000 is representedas a pixel image. Attempting to correctly identify the entire pixelimage may require significant amount of computation power that isunnecessary if the pixel image may be analyzed in portions while stillcorrectly identifying the pixel image based on those analyzed portions.As a result, a convolution layer may be executed dividing the pixelimage into feature maps where each feature map is a smaller portion ofthe pixel image while extracting the data from a portion of the pixelsincluded in the pixel image rather than the data from each pixelincluded in the pixel image.

In this example, the initial image is a 28×28 pixel image. Rather thanapplying the entire 28×28 pixel image to the analog neuromorphic circuit1000 as inputs, 5×5 sections of the 28×28 pixel image are selected suchthat 25 pixels included in each 5×5 section are applied to the analogneuromorphic circuit 1000 as inputs. For example, as shown in FIG. 10,each of the 25 pixels are applied to the analog neuromorphic circuit1000 as the input voltages 440(a-n), where n is equal to 25, and eachcomplement of the 25 pixels are applied to the analog neuromorphiccircuit 1000 as the complemented input voltages 460(a-n), where n isequal to 25. The conductance 490 a and the conductance 490 b for eachpair of the resistive memories 410(a-n) may then be determined asdiscussed in detail above such that the resistive memories 410(a-n)represent a 25-pixel filter that is applied to each 5×5 section of the28×28 pixel image. In doing so, each 5×5 section of the 28×28 pixelimage is multiplied by the 25-pixel filter represented by the resistancevalues of each resistive memory 410(a-n) by simply changing the inputvoltages 440(a-n) and the complemented input voltages 460(a-n) tocorrespond to each 5×5 section of the 28×28 pixel image.

In this example, six feature maps with each feature map including 24×24pixels are generated in parallel by the analog neuromorphic circuit1000. The analog neuromorphic circuit 1000 includes the vertical wires230(a-n), where n is an integer equal to six, with resistive memories410(a-n) positioned on each vertical wire 230(a-n) to correspond to theinput voltages 440(a-n) and the complemented input voltages 460(a-n). Aseach of the 25 pixels for each of the 5×5 sections are applied to theanalog neuromorphic circuit 1000 via the input voltages 440(a-n) and thecomplemented input voltages 460(a-n), the analog neuromorphic circuit1000 generates dot-product operation values 470(a-n), where n is aninteger equal to six, and the complemented dot-product operation values450(a-n), where n is an integer equal to six. Each of the sixdot-product operation values 470(a-n) and the complemented dot-productoperation values 450(a-n) represent the six different feature mapsgenerated in parallel during execution of the feature layer.

Thus, six different 24×24 pixel feature maps are generated by the analogneuromorphic circuit 1000 with each feature map including less pixelsthan the original 28×28 pixel image while incorporating sufficient datafrom the original 28×28 pixel image so that the original 28×28 pixelimage may eventually be identified. The six different 24×24 pixelfeature maps after being generated may then be stored in a digitalstorage layer as the output of the first convolution layer. A datacontroller may be incorporated to reduce the amount of memory requiredto store each of the six different 24×24 pixel feature maps followingthe execution of the first convolution layer. Any type of storage may beincorporated to store the six different 24×24 pixel feature mapsfollowing execution of the first convolution layer that is apparent tothose skilled in the art. Although the above example depicts a 28×28pixel image that is divided into 5×5 pixel portions and applied to a25-pixel filter to generate six feature maps, the analog neuromorphiccircuit 1000 may be modified to handle any size of an initial imagewhile incorporating any size filter to generate any number of featuremaps that is apparent to those skilled in the art.

At step 920, the operational control flow 900 executes a smoothing layerand subsamples the data for each feature map that is stored aftercompleting the first convolution layer in step 910. Each of the featuremaps generated by the analog neuromorphic circuit 1000 and representedas the dot-product operation values 470(a-n) and the complementeddot-product operation values 450(a-n) are then applied as inputs to theanalog neuromorphic circuit 1100 depicted in FIG. 11. The analogneuromorphic circuit 1100 may then execute a smoothing operation where asingle filter is applied to each of the feature maps. In doing so, thepixels included in each of the feature maps may be averaged so that thedata represented by each pixel is averaged. The pixel size of eachfeature map may then be decreased with a subsampling operation where aportion of the averaged pixels for each feature map are selected suchthat the important data of each feature map is carried forward asoutputs of the analog neuromorphic circuit 1100 so that the originalimage may be eventually identified.

The analog neuromorphic circuit 1100 shares many similar features withthe analog neuromorphic processing device 100, the analog neuromorphiccircuit 200, the neural network configuration 300, the analogneuromorphic circuit 400, the output configuration 500, and the analogneuromorphic circuit 1000 therefore, only the differences between theanalog neuromorphic circuit 1100 and the analog neuromorphic circuit1000, the output configuration 500, the analog neuromorphic circuit 400,the analog neuromorphic processing device 100, the analog neuromorphiccircuit 200, and the neural network configuration 300 are to bediscussed in further detail.

For example, six feature maps are generated as dot-product operationvalues 470(a-n) and complemented dot-product operation values 450(a-n)with each feature map including 24×24 pixels. Each of the six featuremaps are then applied as inputs to each individual analog neuromorphiccircuit 1110(a-n), where n is equal to six. Since each of the featuremaps include 24×24 pixels, each feature map is divided into 4×4 sectionsand each 4×4 section is applied as input voltages 440(a-n), where n isan integer equal to four, and complemented input voltages 460(a-n),where n is an integer equal to four. In this example, the first 4×4section of the first feature map is applied as input voltage 440(a 1-n1) and as complemented input voltage 460(a 1-n 1) to the individualanalog neuromorphic circuit 1110 a. Each 4×4 section for each featuremap is applied to the corresponding individual analog neuromorphiccircuit 1110(b-n), where n is equal to 6, in a similar manner.

The smoothing filter applied to each 4×4 section of each feature map maybe executed by each individual analog neuromorphic circuit 1110(a-n) byadjusting the conductance 490 a and the conductance 490 b for each pairof the resistive memories 410(a-n) included in each of the individualanalog neuromorphic circuits 1100(a-n). The conductance 490 a and theconductance 490 b for each resistive memory 410(a-n) included in each ofthe individual analog neuromorphic circuits 1100(a-n) may be adjustedsuch that each of the feature maps are multiplied by the smoothingfilter incorporated into each corresponding individual analogneuromorphic circuit 1110(a-n).

In this example, the conductance 490 a 1 and the conductance 490 b 1 maybe adjusted such that resistance values for each of the resistivememories 410(a 1-n 1) represent the smoothing filter applied by theindividual analog neuromorphic circuit 1110 a to the first feature map.Each of the conductance and conductance for each of the resistivememories included in each of the individual analog neuromorphic circuits1100(b-n) may be adjusted in a similar manner such that eachcorresponding feature map is multiplied by the smoothing filter. In thisexample, the smoothing filter applied to each filter may be determinedaccording to Equation 18,

$\begin{matrix}{k = {\begin{matrix}{1/4} & {1/4} \\{1/4} & {1/4}\end{matrix}.}} & (18)\end{matrix}$

The pixel size of each feature map may also be decreased with asubsampling operation where a portion of the averaged pixels for eachfeature map are selected. In doing so, a portion of the dot-productoperation values 470(a-n) and the complemented dot-product operationvalues 450(a-n) generated by each of the individual analog neuromorphiccircuits 1110(a-n) are selected to be incorporated into a reducedfeature map. For example, the subsampling operation reduces the pixelsize of each feature map by a factor of two in which every otherdot-product operation value 470(a-n) and the complemented dot-productoperation values 450(a-n) generated by each of the individual analogneuromorphic circuits 1110(a-n) are selected to be incorporated into areduced feature map. Each of the selected dot-product operation values470(a-n) and the selected complemented dot-product operation values450(a-n) generated by each of the individual analog neuromorphiccircuits 1110(a-n) may then be stored in a digital storage layer as theoutput of the first smoothing and subsampling layer. Any type of storagemay be incorporated to store the selected dot-product operation values470(a-n) and the selected complemented dot-product operation values450(a-n) generated by each of the individual analog neuromorphiccircuits 1110(a-n) following the execution of the first smoothing andsubsampling layer that is apparent to those skilled in the art.

In this example, every other dot-product operation value 470 a 1 andevery other complemented dot-product operation value 450 n 1 asgenerated by the individual analog neuromorphic circuit 1110 a isselected to be incorporated into a reduced feature map and storedfollowing the selection. The dot-product operation values 470(a-n) andthe complemented dot-product operation values 450(a-n) for each of theindividual analog neuromorphic circuits 1110(b-n) may be selected to beincorporated into the reduced feature map and stored following theselection in a similar manner. Thus, each of the six 24×24 pixel maps isreduced to six 12×12 reduced pixel maps due to the smoothing andsubsampling performed by the analog neuromorphic circuit 1100. In doingso, the pixel size of each feature map is decreased such that theimportant data of each feature map is carried forward and stored asoutputs of the analog neuromorphic circuit 1100 so that the originalimage may be eventually identified.

Although the above example depicts six 24×24 feature maps divided into4×4 pixel portions and applied to a 4×4 pixel filter as depicted inEquation 18 and sampled by a factor of 2, the analog neuromorphiccircuit 1100 may be modified to handle any quantity and any size offeature maps while incorporating any size filter as well as any type offilter and any factor of sampling to generate any number of reducedpixel sets that is apparent to those skilled in the art.

At step 930, the operational control flow 900 executes a secondconvolution layer in parallel by decreasing the reduced pixel mapsgenerated by summing together specified reduced pixel maps based on theselected dot-product operation values 470(a-n) and the complementeddot-product operation values 450(a-n) stored in step 920. Each of thereduced pixel maps generated by the analog neuromorphic circuit 1100 andrepresented by the dot-product operation values 470(a-n) and thecomplemented dot-product operation values 450(a-n) are then applied asinputs to the analog neuromorphic circuit 1200 as depicted in FIG. 12.The analog neuromorphic circuit 1200 may then execute a secondconvolution layer where each of the reduced pixel maps applied to theanalog neuromorphic circuit 1200 are multiplied in parallel by differentkernels with each kernel represented by the resistance values of thecorresponding resistive memories 410(a-n) associated with each verticalwire 230(a-n).

The amount of feature maps generated by the second convolution layer asrepresented by the dot-product operation values 470(a-n) and thecomplemented dot-product operation values 450(a-n) of the analogneuromorphic circuit 1200 may then be reduced. Each of the feature mapsrepresented by the dot-product operation values 470(a-n) and thecomplemented dot-product operation values 450(a-n) may be groupedtogether into groups and then each of the feature maps included in acorresponding group may be summed together to generate a grouped featuremap that represents the feature maps included in each group and therebyreduce the amount of feature maps generated by the analog neuromorphiccircuit 1200. The feature maps generated by the analog neuromorphiccircuit 1200 after executing the second convolution layer may also havea decreased pixel size as compared to the feature maps initiallyprovided as inputs to the analog neuromorphic circuit 1200 such that theimportant data of each feature map is carried forward as outputs of theanalog neuromorphic circuit 1200 so that the original image may beeventually identified.

The analog neuromorphic configuration 1200 shares many similar featureswith the analog neuromorphic processing device 100, the analogneuromorphic circuit 200, the neural network configuration 300, theanalog neuromorphic circuit 400, the output configuration 500, theanalog neuromorphic circuit 1000, and the analog neuromorphic circuit1100, therefore, only the differences between the analog neuromorphiccircuit 1200 and the analog neuromorphic circuit 1100, the analogneuromorphic circuit 1000, the output configuration 500, the analogneuromorphic circuit 400, the analog neuromorphic processing device 100,the analog neuromorphic circuit 200, and the neural networkconfiguration 300 are to be discussed in further detail.

For example, as noted above, six reduced pixel maps are generated by theanalog neuromorphic circuit 1100 as dot-product operation values470(a-n) and complemented dot-product operation values 450(a-n) witheach reduced pixel map including 12×12 pixels. Each of the six reducedpixel maps are then applied as inputs to the analog neuromorphic circuit1200(a-n), where n is equal to six. Since each of the six reduced pixelmaps include 12×12 pixels, each reduced pixel map is applied as inputvoltages 440(a-n), where n is an integer equal to twelve, andcomplemented input voltages 460(a-n), where n is an integer equal toequal to twelve. In this example, the first reduced pixel map is appliedas input voltages 440(a 1-n 1) and as complemented input voltages 460(a1-n 1) to the analog neuromorphic circuit 1200. Each reduced pixel mapis applied to the analog neuromorphic circuit 1200 in a similar manneras shown where the sixth reduced pixel map is applied as input voltages440(an-nn), where n is an integer equal to twelve, and as complementedinput voltages 460(an-nn), where n is an integer equal to twelve.

Different kernels are then applied to each of the reduced pixel mapsthat are applied as inputs to the analog neuromorphic circuit 1200. Eachkernel is represented by resistance values associated with each of thecorresponding resistive memories 410(a-n) included in a correspondingvertical wire 230(a-n). For example, a first kernel is represented bythe resistance values associated with resistive memories 410(a 1-f 1)that is applied to the first reduced pixel map applied to the analogneuromorphic circuit 1200 as input voltages 440(a 1-n 1) and ascomplemented input voltages 460(a 1-n 1). As can be seen, the firstkernel includes each of the resistive memories 410(a 1-f 1) that ispositioned on the vertical wire 230 a. In this example, a twelfth kernelis represented by resistance values associated with each of thecorresponding resistive memories 410(an-fn), where n is an integer equalto twelve. As can be seen, the twelfth kernel includes each of theresistive memories 410(an-fn) that is positioned on the vertical wire230 n, where n is an integer equal to twelve. The positioning of each ofthe resistive memories 410(a-n) on a corresponding vertical wire230(a-n) to establish each of the different kernels as well as thesimultaneous application of each of the different reduced pixel maps asinputs to the analog neuromorphic circuit 1200 enables the execution ofthe second convolution layer where each of the different feature mapsare simultaneously multiplied by each of the different kernels to beexecuted in parallel.

Each of the kernels applied to each of the reduced pixel maps may beexecuted by the analog neuromorphic circuit 1200 by adjusting each ofthe conductance values 490(a 1-an), where n is an integer equal totwelve, and each of the conductance values 490(an-bn), where n is aninteger equal to twelve, for each pair of resistive memories applied toeach of the reduced pixel maps. The conductance values 490(a 1-an) andthe conductance values 490(an-bn) for each of the correspondingresistive memories applied to each of the corresponding reduced pixelmaps such that each of the reduced pixel maps are multiplied by each ofthe different kernels as positioned on each of the correspondingvertical wires 230(a-n).

In this example, the conductance 490 a 1 and the conductance 490 b 1 maybe adjusted such that resistance values for each of the resistivememories 410(a 1-fn) that represent each of the different kernelsapplied by the analog neuromorphic circuit 1200 to the first reducedpixel map. Each of the conductance values 490(a 2-an) and conductancevalues 490(b 2-bn) for each of the resistive memories included in theanalog neuromorphic circuit 1200 may be adjusted in a similar mannersuch that each corresponding reduced pixel map is multiplied by each ofthe different filters.

The application of different kernels to each of the different reducedpixel maps via the analog neuromorphic circuit 1200 may result in asignificant amount of output feature maps with a significant portion ofthose output feature maps being unnecessary to eventually correctlyidentify the pixel image. For example, the six different reduced pixelmaps applied as inputs to the analog neuromorphic circuit 1200 that aresimultaneously multiplied with twelve different kernels result in 72different convolutions that are performed generating 72 different outputfeature maps by the analog neuromorphic circuit 1200. However, 72different output feature maps are unnecessary to eventually correctlyidentify the pixel image.

Rather than generate significant amount of unnecessary output featuremaps, each of the output feature maps may be grouped together intodifferent groups where each of the feature maps included in a group maybe summed together to generate a single output feature map that isrepresentative of the group. Each group may then have a single groupedoutput feature map associated with the corresponding group such thateach of the grouped output feature maps may then be generated as outputsof the analog neuromorphic circuit 1200. Thus, the significant quantityof originally generated output feature maps may be significantly reducedto the grouped output feature maps. Each of the single grouped featuremap associated with the corresponding group may then be stored in adigital storage layer as the output of the second convolution layer. Anytype of storage may be incorporated to store the single grouped featuremaps following the execution of the second convolution layer that isapparent to those skilled in the art.

For example, each of the 72 different output feature maps generated bythe simultaneous multiplication of the six different feature maps withthe twelve different kernels may be grouped together such that six ofthe 72 output feature maps are included in a single group. Thus, twelvedifferent groups may be generated by grouping the 72 output feature mapsinto groups of six. Each of the output feature maps included in a groupmay then be summed together to generate the grouped output feature mapthat is representative of each of the six output feature maps includedin the group such that each of the twelve different groups may generatetwelve different grouped output feature maps and then stored.

As noted above, each of the twelve different kernels are positioned on acorresponding vertical wire 230(a-n) resulting in twelve differentvertical wires 230(a-n) with each of the twelve different vertical wires230(a-n) generating a corresponding output value represented by thedot-product operation values 470(a-n), where n is an integer equal totwelve, and complemented dot-product operation values 450(a-n), where nis an integer equal to twelve. As a result, each of the twelve differentgrouped output feature maps generated by the grouping the 72 differentoutput feature maps into groups of six may be represented by each of thecorresponding the dot-product operation values 470(a-n) and complementeddot-product operation values 450(a-n) generated by each of the verticalwires 230(a-n).

In this example, the grouping of the output feature maps may beimplemented in Equation 19 where i=1, . . . N and N is equal to thenumber of input maps (N=6). Likewise, j=1, . . . , M and M is equal tothe number of output maps (M=12). Each X_(i) denotes an entire inputarray containing 25 elements, and each K_(ij) denotes an entire kernelarray containing 25 elements.v _(j)=(X ₁ *K _(1j))+(X _(i) *K _(ij))+ . . . +(X _(N) *K _(Nj)).  (19)Since convolution (denoted by *) is a linear operation, the equation maybe rearranged so that the entire process may be completed in a singlecrossbar column as in equation 20,v _(j)=[X ₁ , . . . X _(i) , . . . ,X _(N)]*[K _(1j) , . . . ,K _(ij) ,. . . K _(Nj)].  (20)Thus, the outputs of the analog neuromorphic circuit 1200 afterexecuting the second convolution layer may include an additional set offeature maps each with a decreased pixel size from the feature mapsinitially applied as inputs to the analog neuromorphic circuit 1200. Inthis example, the outputs of the analog neuromorphic circuit 1200 afterexecuting the second convolution layer include twelve 8×8 pixel featuremaps that are reduced from the initial 12×12 pixel feature maps appliedas inputs to the analog neuromorphic circuit 1200. In doing so, theimportant data of each feature map is carried forward as outputs of theanalog neuromorphic circuit 1200 and stored in a digital storage layerso that the original image may be eventually identified.

Although the above example depicts six 12×12 feature maps simultaneouslyinputted to the analog neuromorphic circuit 1200 and simultaneouslymultiplied by twelve different kernels generating 72 different outputfeature maps that are then grouped into twelve different grouped outputfeature maps each including six different output feature maps, theanalog neuromorphic circuit 1200 may be modified to handle any quantityand any size of feature maps while incorporating any amount of kernelsas well as grouping output feature maps into any quantity of groupsincluding any quantity of output feature maps to generate any quantityof grouped output feature maps that is apparent to those skilled in theart.

At step 940, the operational control flow 900 executes a secondsmoothing layer and subsamples the data for each feature map based onthe grouped output feature maps stored in step 930. Similar to step 920,an analog neuromorphic configuration similar to the analog neuromorphiccircuit 1100 may be implemented to execute the smoothing layer andsubsampling executed in step 920. However, in this example, rather thanhaving six feature maps applied as inputs to each individual analogneuromorphic circuit similar the individual analog neuromorphic circuits1110(a-n), twelve feature maps as were outputted by the analogneuromorphic circuit 1200 in step 930 are applied as inputs such that nis equal to twelve rather than six as in step 920. Rather than each ofthe feature maps including 24×24 pixels as in step 920, each of thefeature maps as outputted by the analog neuromorphic circuit 1200 instep 930 include 8×8 pixels. Each of the 8×8 pixel feature maps are thenreduced and stored in a similar manner as the smoothing and subsamplingexecuted in step 920 where each of the 8×8 pixel maps are reduced totwelve different 4×4 pixel maps that are then stored in a digitalstorage layer.

At step 950, the operational control flow 900 executes a classificationlayer to classify the resulting feature maps stored in step 940. Asnoted above with regards to step 940, several feature maps are generatedas outputs after the second smoothing layer and subsampling is executedwith each of those feature maps including a pixel size that has beenreduced further. The feature maps generated by step 940 may include areduced pixel size such that each of those feature maps may be easilyapplied as inputs to an analog neuromorphic circuit similar to that ofthe analog neuromorphic circuit 400 shown in FIG. 4. However, each ofthe feature maps applied as inputs to the analog neuromorphic circuit400 may still include sufficient information such that the dot-productoperation values 470(a-n) and complemented dot-product operation values450(a-n) adequately identify the original pixel image applied to theanalog neuromorphic circuit 1000 depicted in FIG. 10 in step 910.

Each of the values included in each of the feature maps may be appliedto the analog neuromorphic circuit 400 as input voltages 440(a-n) andcomplemented input voltages 460(a-n). For example, twelve different 4×4pixel feature maps as generated as outputs in step 940 may be applied tothe analog neuromorphic circuit 400. In such an example, each of thetwelve different 4×4 pixel feature maps include sixteen different valuesresulting in 192 input values to the analog neuromorphic circuit 400 asinput voltages 440(a-n), where n is an integer equal to 192, andcomplemented input voltages 460(a-n), where n is an integer equal to192.

The analog neuromorphic circuit 400 may then include a quantity ofvertical wires 230(a-n) that correspond to the quantity of outputs thatthe analog neuromorphic circuit 400 may generate. For example, theinitial 24×24 pixel image applied to the analog neuromorphic circuit1000 in step 910 may be an image of a handwritten digit included in theMixed National Institute of Standards and Technology (MNIST) databasewhich includes images of handwritten digits ranging from 0 through 9. Insuch an example, the analog neuromorphic circuit may include tendifferent vertical wires 230(a-n), where n is an integer equal to ten,to correspond to the ten possible outputs that may be generatedcorresponding to the 10 different handwritten digits (0-9) that may beapplied as the initial 24×24 pixel image.

Each of the resistance values for each of the corresponding resistivememories included in the analog neuromorphic circuit 400 may be adjustedbased on the conductance 490 a and the conductance 490 b as discussed indetail above with regard to FIG. 4. As each of the 192 values associatedwith each of the twelve 4×4 pixel maps are applied as inputs to theanalog neuromorphic circuit 400, each of the dot-product operationvalues 470(a-n), where n is an integer equal to ten, and complementeddot-product operation values 450(a-n), where n is an integer equal toten, may correspond to the appropriate handwritten digit applied as theinitial 24×24 pixel image in step 910. For example, the initial 24×24pixel image applied to the analog neuromorphic circuit in step 910 maybe the handwritten image of “1”. The dot-product operation value 470 band the complemented dot-product operation value 450 b may output a“high” voltage signal while the remaining dot-product operation values470(a, c-n) and complemented dot-product operation values 450(a, c-n)depict a “low” voltage signal thus recognizing that the initial 24×24pixel image applied to the analog neuromorphic circuit in step 910 isthe handwritten image of “1”.

As noted above regarding FIGS. 9 and 10, the initial image that isapplied to the analog neuromorphic circuit 1000 as discussed regardingstep 910, is applied as inputs to the analog neuromorphic circuit 1000in sections to generate the feature maps as outputs of the analogneuromorphic circuit 1000. As discussed in the example above, theinitial 28×28 pixel image is applied to the analog neuromorphic circuit1000 as inputs in 5×5 pixel sections to generate the six different 24×24pixel feature maps as outputs of the analog neuromorphic circuit 1000.Further as discussed in the example above, each of the six different24×24 pixel feature maps generated as outputs of the analog neuromorphiccircuit 1000 may then be stored in a digital storage layer of the firstconvolution layer until the first smoothing layer and subsampling of thesix different feature maps is executed in step 920.

Rather than break the initial image that is applied to the analogneuromorphic circuit 1000 as inputs into sections and then digitallystore each of the feature maps generated as outputs, the entire initialimage may be applied to the analog neuromorphic circuit as inputswithout having to break the initial image into sections to generate eachof the feature maps. In doing so, multiple output feature maps may begenerated in a single processing cycle rather than having each of thedifferent sections of the initial image be applied to the kernelrepresented by the resistive memories 410(a-n) included in the analogneuromorphic circuit 1000. The application of the initial image to theanalog neuromorphic circuit in a single processing cycle withoutapplying the initial image in sections may also eliminate the need todigitally store each of the output feature maps until the firstsmoothing layer and subsampling of the output feature maps is executed.

As discussed in the example above regarding the initial embodiment ofstep 910 and shown in FIG. 10, the analog neuromorphic circuit 1000 is a51×6 resistive memory 410(a-n) crossbar where κ×5 pixel sections of the28×28 pixel initial image is applied as input voltages 440(a-n) andcomplemented input voltages 460(a-n) in order to generate the sixdifferent 24×24 pixel feature maps. In doing so, six different 5×5convolution kernels may be represented by the 51×6 resistive memory410(a-n) crossbar and applied to each of the 5×5 pixel sections of the28×28 pixel initial image to generate the six different 24×24 pixelfeature maps. However, in a first alternative embodiment regarding step910, the entire 28×28 pixel initial image may be applied to the analogneuromorphic circuit to generate the six different 24×24 feature maps.In doing so, six different 5×5 convolution kernels may be represented bysix different crossbars with each crossbar including 1569×576 ofresistive memories and applied to the entire 28×28 pixel initial imageto generate the six different 24×24 feature maps in a single cycle.

In executing the first alternative embodiment for step 910, the mappingof the convolution kernels onto the analog neuromorphic circuit differsfrom the mapping of the convolution kernels onto the analog neuromorphiccircuit 1000 as discussed in the initial embodiment of step 910. Themapping of the convolution kernels onto the analog neuromorphic circuit1000 as discussed in initial embodiment of step 910 is discussed in thedetailed example above regarding a 3×3 kernel where the 3×3 kernelexample is used for simplicity in discussion purposes. The 3×3 kernel inEquation 21 is stored in a column of the analog neuromorphic circuit sothat each input value is aligned with the correct kernel value:

$\begin{matrix}{k_{ex} = {\begin{bmatrix}0.1 & {- 0.2} & 0.3 \\{- 0.4} & 0.5 & {- 0.6} \\0.7 & {- 0.8} & 0.9\end{bmatrix}->{\begin{pmatrix}0.9 \\{- 0.6} \\0.3 \\{- 0.8} \\0.5 \\{- 0.2} \\0.7 \\{- 0.4} \\0.1\end{pmatrix}->{\overset{k_{ex}^{+}}{\begin{pmatrix}0.9 \\0 \\0.3 \\0 \\0.5 \\0 \\0.7 \\0 \\0.1\end{pmatrix}}->{\overset{k_{ex}^{-}}{\begin{pmatrix}0 \\0.6 \\0 \\0.8 \\0 \\0.2 \\0 \\0.4 \\0\end{pmatrix}}.}}}}} & (21)\end{matrix}$

Furthermore, the kernel array is converted into two arrays, k_(ex)+ andk_(ex) ⁻, so that the analog neuromorphic circuit may account forkernels that have both positive and negative values. Similarly, the 3×3pixel image as discussed in the detailed example above may be convertedinto two arrays of identical values with reverse sign as shown inEquation 22:

$\begin{matrix}{x_{ex} = {\begin{bmatrix}0 & 0.5 & 0.3 \\0.5 & 0.8 & 0.5 \\0 & 0.5 & 0\end{bmatrix}->{\begin{pmatrix}0 \\0.5 \\0.3 \\0.5 \\0.8 \\0.5 \\0 \\0.5 \\0\end{pmatrix}->{\overset{x_{ex}}{\begin{pmatrix}0 \\0.5 \\0.3 \\0.5 \\0.8 \\0.5 \\0 \\0.5 \\0\end{pmatrix}}{\overset{- x_{ex}}{\begin{pmatrix}0 \\{- 0.5} \\{- 0.3} \\{- 0.5} \\{- 0.8} \\{- 0.5} \\0 \\{- 0.5} \\0\end{pmatrix}}.}}}}} & (22)\end{matrix}$

However, regarding the first alternative embodiment to step 910, each ofthe convolution kernels may be expanded into large sparse matrices ascompared Equation 21 above. For example, the 3×3 convolution kernel,k_(ex) as shown in Equation 22 used for simplicity in discussionpurposes, displays how k_(ex) may be converted into k_(exp) ⁺ andk_(exp) ⁻ to account for kernels that have positive and negative values:

$\begin{matrix}{k_{ex} = {\begin{bmatrix}0.1 & {- 0.2} & 0.3 \\{- 0.4} & 0.5 & {- 0.6} \\0.7 & {- 0.8} & 0.9\end{bmatrix}->{\overset{k_{{ex}\; p}^{+}}{\begin{pmatrix}0.9 & 0 & 0 & 0 \\0 & 0.9 & 0 & 0 \\0.3 & 0 & 0 & 0 \\0 & 0.3 & 0 & 0 \\0 & 0 & 0.9 & 0 \\0.5 & 0 & 0 & 0.9 \\0 & 0.5 & 0.3 & 0 \\0 & 0 & 0 & 0.3 \\0.7 & 0 & 0 & 0 \\0 & 0.7 & 0.5 & 0 \\0.1 & 0 & 0 & 0 \\0 & 0.1 & 0 & 0 \\0 & 0 & 0.7 & 0 \\0 & 0 & 0 & 0.7 \\0 & 0 & 0.1 & 0 \\0 & 0 & 0 & 0.1\end{pmatrix}}{\overset{k_{e\; x\; p}^{-}}{\begin{pmatrix}0 & 0 & 0 & 0 \\0.6 & 0 & 0 & 0 \\0 & 0.6 & 0 & 0 \\0 & 0 & 0 & 0 \\0.8 & 0 & 0 & 0 \\0 & 0.8 & 0.6 & 0 \\0.2 & 0 & 0 & 0.6 \\0 & 0.2 & 0 & 0 \\0 & 0 & 0.8 & 0 \\0.4 & 0 & 0 & 0.8 \\0 & 0.4 & 0.2 & 0 \\0 & 0 & 0 & 0.2 \\0 & 0 & 0 & 0 \\0 & 0 & 0.4 & 0 \\0 & 0 & 0 & 0.4 \\0 & 0 & 0 & 0\end{pmatrix}}.}}}} & (23)\end{matrix}$Further for this smaller scale example for simplicity in discussionpurposes, the initial image, x_(ex), is a 4×4 pixel image that may beconverted into x_(exp) and −x_(exp) which are vector versions of theinput image, x_(ex), as shown in Equation 24:

$\begin{matrix}{x_{ex} = {\begin{pmatrix}x_{1} & \ldots & x_{4} \\\vdots & \ddots & \vdots \\x_{13} & \ldots & x_{16}\end{pmatrix}->{\overset{x_{e\;{xp}}}{\begin{pmatrix}x_{1} \\\vdots \\x_{16}\end{pmatrix}}{\overset{- x_{e\;{xp}}}{\begin{pmatrix}{- x_{1}} \\\vdots \\{- x_{16}}\end{pmatrix}}.}}}} & (24)\end{matrix}$The initial image, x_(ex), includes 16 elements that is then convertedinto vector versions x_(exp) and −x_(exp), which are then multiplied byeach kernel matrix, k_(exp) ⁺ and k_(exp) ⁻, that includes 16 rows,respectively. Since the convolution kernel, k_(ex), has the dimensionsof 3×3 and the initial image x_(ex), has the dimensions of 4×4, theresulting output feature map may have the dimensions of 2×2 resulting inthe kernel matrices of k_(exp) ⁺ and k_(exp) ⁻ having 4 columnscorresponding to each output value.

Thus, the mapping of the convolution kernels onto the analogneuromorphic circuit regarding the first alternative embodiment to step910 as shown by Equation 23 depicts several different column outputsdepicted by k_(exp) ⁺ and k_(exp) ⁻ as compared to the single columnoutput of k_(exp)+ and k_(exp) ⁻ as shown by Equation 21 for the initialembodiment of step 910. Rather than having a single column output ofk_(exp) ⁺ and k_(exp) ⁻ for the initial embodiment of step 910 wheredifferent sections of the initial image is applied to the analogneuromorphic circuit 1000 to generate the different feature maps, theseveral different column outputs of k_(exp)+ and k_(exp) ⁻ for the firstalternative embodiment of step 910 enables the entire initial image tobe applied to the analog neuromorphic circuit to generate differentfeature maps in a single cycle. As noted above, although the aboveexample depicts a 28×28 pixel image that is divided into 5×5 pixelportions and applied to a 25-pixel filter to generate six feature maps,the analog neuromorphic circuit may be modified to handle any size of aninitial image while incorporating any size filter to generate any numberof feature maps that is apparent to those skilled in the art.

As noted above regarding FIGS. 9 and 11, each of the different featuremaps is applied to the analog neuromorphic circuit 1100 as discussedregarding step 920, is applied as inputs to the analog neuromorphiccircuit 1100 in sections as well as a selection of output values of theanalog neuromorphic circuit 1100 in executing the subsampling togenerate the reduced pixel maps as outputs of the analog neuromorphiccircuit 1100. As discussed in the example above, each of the six 24×24pixel feature maps is applied to a corresponding individual analogneuromorphic circuit 1110(a-n) as inputs in 4×4 pixel sections.

For example, the first 24×24 pixel feature map is applied to theindividual analog neuromorphic circuit 1110 a in 4×4 sections, thesecond 24×24 pixel feature map is applied to the individual analogneuromorphic circuit 1110 b in 4×4 sections and so on. In doing so, thefirst smoothing layer may be applied to each of the six 24×24 pixelfeature maps in parallel. Further as discussed in the example above, thepixel size of feature maps may be decreased with a subsampling operationwhere a portion of the averaged pixels for each feature map areselected. Further as discussed in the example above, each of the sixdifferent 12×12 reduced pixel maps generated as outputs of the analogneuromorphic circuit 1100 may then be stored in a digital storage layerof the first smoothing layer and subsampling until the secondconvolution layer of the six different reduced pixel maps is executed instep 930.

Rather than break each of the feature maps that are applied to theanalog neuromorphic circuit 1100 as inputs into sections and thendigitally store each of the reduced pixel maps generated as outputs aswell as perform the subsampling in selecting a portion of the outputvalues, the entire image may be applied to the analog neuromorphiccircuit 1300 as depicted in FIG. 13 as inputs without having to breakeach of the feature maps into sections to generate each of the reducedpixel maps. Further, rather than generating feature maps with valuesthat are then selected to generate the reduced pixel maps viasubsampling, the columns of the convolution kernel matrixes, k_(exp) ⁺and k_(exp) ⁻ in Equation 23, as generated in the first convolutionlayer presented in the first alternative embodiment of step 910 thatwould be removed via the subsampling are simply not carried over andapplied to the analog neuromorphic circuit 1300 as inputs. As the outputvalues corresponding to those inputs to the analog neuromorphic circuit1300 would be eliminated regardless in a subsampling operation. Theapplication of each feature map to the analog neuromorphic circuit 1300in a single processing cycle without applying the feature maps insections as well as not having to perform the subsampling via selectingthe output values of the analog neuromorphic circuit 1300 may alsoeliminate the need to digitally store each of the output feature mapsuntil the second convolution layer of the reduced pixel maps isexecuted.

The analog neuromorphic circuit 1300 shares many similar features withthe analog neuromorphic processing device 100, the analog neuromorphiccircuit 200, the neural network configuration 300, the analogneuromorphic circuit 400, the output configuration 500, the analogneuromorphic circuit 1000, the analog neuromorphic circuit 1100, and theanalog neuromorphic circuit 1200, therefore, only the differencesbetween the analog neuromorphic circuit 1300 and the analog neuromorphiccircuit 1200, the analog neuromorphic circuit 1100, the analogneuromorphic circuit 1000, the output configuration 500, the analogneuromorphic circuit 400, the analog neuromorphic processing device 100,the analog neuromorphic circuit 200, and the neural networkconfiguration 300 are to be discussed in further detail.

As discussed in the example above regarding the initial embodiment ofstep 920 and shown in FIG. 11, the analog neuromorphic circuit 1100includes six different individual analog neuromorphic circuits1110(a-n). Each 4×4 pixel section of each corresponding 24×24 pixelfeature map is applied as input voltages 440(a 1-nn) and complementedinput voltages 460(a 1-nn) to the corresponding individual analogneuromorphic circuit 1110(a-n) in order to generate the six different12×12 reduced pixel maps. In doing so, each of the six smoothing filtersmay be represented via the respective resistive memories 410(a 1-nn)that are included in each corresponding individual analog neuromorphiccircuit 1110(a-n).

However, in a first alternative embodiment regarding step 920, each ofthe entire 24×24 pixel feature images may be applied to the analogneuromorphic circuit 1300 to generate the six different 12×12 reducedpixel maps. In doing so, each of the six smoothing filters may berepresented by six different individual analog neuromorphic circuits1310(a-n), where n is an integer equal to six in this example, with eachindividual analog neuromorphic circuit 1310(a-n) including 1153×144resistive memories. Each of the 24×24 pixel feature maps are thenreduced by eliminating the unwanted columns from the columns of theconvolution kernel matrixes, k_(exp) ⁺ and k_(exp) ⁻ in Equation 23,generated in the first convolution layer presented in the firstalternative embodiment of step 910 and the remaining values are thenapplied as inputs to each corresponding individual analog neuromorphiccircuit 1310(a-n) to generate the six different 12×12 reduced pixel mapsin a single cycle.

Although the above example depicts six 24×24 feature maps divided into4×4 pixel portions and applied to a 4×4 pixel filter as depicted inEquation 18 and sampled by a factor of 2, the analog neuromorphicconfiguration 1300 may be modified to handle any quantity and any sizeof feature maps while incorporating any size filter as well as any typeof filter and any factor of sampling to generate any number of reducedpixel sets that is apparent to those skilled in the art.

As noted above regarding FIGS. 9 and 12, each of the different reducedpixel maps is applied to the analog neuromorphic circuit 1200 asdiscussed regarding the initial embodiment of step 930, is applied asinputs to the analog neuromorphic circuit 1200 in sections. As discussedin the example above, each of the six 12×12 reduced pixel maps isapplied to the analog neuromorphic circuit 1200 as inputs in 5×5 pixelsections.

Rather than break each of the reduced pixel maps that are applied to theanalog neuromorphic circuit 1200 as inputs into sections and thendigitally store each of the output feature maps generated as outputs,the entire reduced pixel map may be applied to the analog neuromorphiccircuit 1400 as depicted in FIG. 14 as inputs without having to breakeach of the reduced pixel maps into sections to generate each of theoutput feature maps. In a first alternative embodiment regarding step930, each of the entire 12×12 pixel reduced pixel maps may be applied tothe analog neuromorphic circuit 1400 and grouped according to generatetwelve 4×4 pixel output feature maps. In doing so, each of the differenttwelve different kernels may be represented by twelve differentindividual analog neuromorphic circuits 1410(a-n), where n is an integerequal to twelve in this example, with each individual analogneuromorphic circuit 1410(a-n) including 288×64 resistive memories. Theapplication of each feature map to the analog neuromorphic circuit 1400in a single processing cycle without applying the reduced pixel maps insections may also eliminate the need to digitally store each of theoutput feature maps until the second smoothing layer is executed.

The analog neuromorphic circuit 1400 shares many similar features withthe analog neuromorphic processing device 100, the analog neuromorphiccircuit 200, the neural network configuration 300, the analogneuromorphic circuit 400, the output configuration 500, the analogneuromorphic circuit 1000, the analog neuromorphic circuit 1100, theanalog neuromorphic circuit 1200, and the analog neuromorphic circuit1300, therefore, only the differences between the analog neuromorphiccircuit 1400 and the analog neuromorphic circuit 1300, the analogneuromorphic circuit 1200, the analog neuromorphic circuit 1100, theanalog neuromorphic circuit 1000, the output configuration 500, theanalog neuromorphic circuit 400, the analog neuromorphic processingdevice 100, the analog neuromorphic circuit 200, and the neural networkconfiguration 300 are to be discussed in further detail.

Although the above example depicts six 12×12 reduced pixel mapssimultaneously inputted to the analog neuromorphic circuit 1400 andsimultaneously multiplied by twelve different kernels generating 72different output feature maps that are then grouped into twelvedifferent grouped output feature maps each including six differentoutput feature maps, the analog neuromorphic circuit 1400 may bemodified to handle any quantity and any size of feature maps whileincorporating any amount of kernels as well as grouping output featuremaps into any quantity of groups including any quantity of outputfeature maps to generate any quantity of grouped output feature mapsthat is apparent to those skilled in the art.

As noted above regarding FIGS. 9, 12, and 13 the images that are appliedto the analog neuromorphic circuits as inputs are applied without havingto break the applied images into sections thereby eliminating the needfor digital storage. In doing so, analog neuromorphic circuits withincreased size regarding the quantity of resistive memories included inthe analog neuromorphic circuits are implemented in order to accommodatethe images being applied as inputs without having to break the appliedimages into sections. However, there may be applications where theimages that are to be input into the analog neuromorphic circuits aresignificantly large with a significant quantity of resistive memorieswhere incorporating such large analog neuromorphic circuits may resultin error that exceeds the requirements of the design. Thus, thesignificantly large analog neuromorphic circuits may then be broken downinto the horizontal and/or vertical direction to generate smaller analogneuromorphic circuits to simplify the analog neuromorphic circuits todecrease the error while still eliminating the need for digital storage.

FIG. 15 depicts an analog neuromorphic circuit 1500 where the initialanalog neuromorphic circuit is broken down horizontally into smalleranalog neuromorphic circuits 1510 a and 1510 n, where n is an integerequal to or greater than one. In splitting the initial analogneuromorphic circuit into the smaller analog neuromorphic circuits1510(a-n) in the horizontal direction, the inputs applied to the initialanalog neuromorphic circuit may be applied to the smaller analogneuromorphic circuits 1510(a-n) simultaneously with each of thedifferent smaller analog neuromorphic circuits 1510(a-n) generating adifferent set of output values. For example, as depicted in the 200×200resistive memories of the smaller analog neuromorphic circuits 1510(a-n)of FIG. 15, one hundred positive input values and one hundred negativeinput values are applied to both the smaller analog neuromorphic circuit1510 a and the smaller analog neuromorphic circuit 1510 nsimultaneously. However, the two hundred output values for the smalleranalog neuromorphic circuit 1510 a differ from the two hundred outputvalues for the smaller analog neuromorphic circuit 1510 n. Although theabove example depicts smaller analog neuromorphic circuits depicted with200×200 resistive memories where the initial analog neuromorphic circuitis broken down horizontally into the smaller analog neuromorphiccircuits, any quantity and any size of smaller analog neuromorphiccircuits may be implemented in breaking down the initial neuromorphiccircuit horizontally that is apparent to those skilled in the art.

The analog neuromorphic circuit 1500 shares many similar features withthe analog neuromorphic processing device 100, the analog neuromorphiccircuit 200, the neural network configuration 300, the analogneuromorphic circuit 400, the output configuration 500, the analogneuromorphic circuit 1000, the analog neuromorphic circuit 1100, theanalog neuromorphic circuit 1200, the analog neuromorphic circuit 1300,and the analog neuromorphic circuit 1400 therefore, only the differencesbetween the analog neuromorphic circuit 1500 and the analog neuromorphiccircuit 1400, the analog neuromorphic circuit 1300, the analogneuromorphic circuit 1200, the analog neuromorphic circuit 1100, theanalog neuromorphic circuit 1000, the output configuration 500, theanalog neuromorphic circuit 400, the analog neuromorphic processingdevice 100, the analog neuromorphic circuit 200, and the neural networkconfiguration 300 are to be discussed in further detail.

FIG. 16 depicts an analog neuromorphic circuit 1600 where the initialanalog neuromorphic circuit is broken down vertically into smalleranalog neuromorphic circuits 1610 a and 1610 n, where n is an integerequal to or greater than one. The initial analog neuromorphic circuitmay be broken down vertically into the smaller analog neuromorphiccircuits 1610(a-n) when the number of inputs for the initial analogneuromorphic circuit spans larger than the amount of inputs for each ofthe smaller analog neuromorphic circuits 1610(a-n). However, the analogneuromorphic circuit 1000 depicted in FIG. 10 enables the flexibility tobreak down the initial analog neuromorphic circuit vertically into thesmaller analog neuromorphic circuits 1610(a-n). For example, an initialimage of 784 pixels may be applied to the smaller analog neuromorphiccircuits 1610(a-n). However, the size of each of the smaller analogneuromorphic circuits 1610(a-n) is a 200×200 resistive memoryconfiguration thereby limiting the quantity of inputs that may beapplied to each of the smaller analog neuromorphic circuits 1610(a-n) totwo hundred inputs. In such an example, seven different smaller analogneuromorphic circuits 1610(a-n) may be implemented in order toaccommodate the 784-pixel initial image. Each of the smaller analogneuromorphic circuits 1610(a-n) may then generate two hundred differentoutputs.

The analog neuromorphic circuit 1600 shares many similar features withthe analog neuromorphic processing device 100, the analog neuromorphiccircuit 200, the neural network configuration 300, the analogneuromorphic circuit 400, the output configuration 500, the analogneuromorphic circuit 1000, the analog neuromorphic circuit 1100, theanalog neuromorphic circuit 1200, the analog neuromorphic circuit 1300,the analog neuromorphic circuit 1400, and the analog neuromorphiccircuit 1500, therefore, only the differences between the analogneuromorphic circuit 1600 and the analog neuromorphic circuit 1500, theanalog neuromorphic circuit 1400, the analog neuromorphic circuit 1300,the analog neuromorphic circuit 1200, the analog neuromorphic circuit1100, the analog neuromorphic circuit 1000, the output configuration500, the analog neuromorphic circuit 400, the analog neuromorphicprocessing device 100, the analog neuromorphic circuit 200, and theneural network configuration 300 are to be discussed in further detail.

In such an embodiment, the smaller analog neuromorphic circuits1610(a-n) that are incorporated into the subsampling layer may notincorporate the approximate sigmoid function as provided in Equations15-17 while the convolution layer does incorporate the approximatesigmoid function as provided in Equations 15-17. In doing so, theapproximate sigmoid function as provided in Equations 15-17 may beremoved from the smaller analog neuromorphic circuits 1610(a-n) that areincorporated into the subsampling layer by setting m=1 to and b=0. Thesmaller analog neuromorphic circuits 1610(a-n) that are incorporatedinto the subsampling layer may then be implemented using Equation 25:y=Σ _(i=1) ^(N) x _(i)σ_(i) +b.  (25)The smaller analog neuromorphic circuits 1610(a-n) that are incorporatedinto the convolution layer may then be implemented using Equation 26where θ denotes the approximate sigmoid function in discussed above inEquation 17:y=θ(Σ_(i=1) ^(N) x _(i)σ_(i) +b).  (26)

Each of the smaller analog neuromorphic circuits 1610(a-n) that do notimplement the approximate sigmoid function may have each of theirrespective outputs provided to a corresponding output analogneuromorphic circuit 1620(a-n), where n is an integer that correspondsto the quantity of smaller analog neuromorphic circuits 1610(a-n). Forexample, the two hundred outputs generated by the smaller analogneuromorphic circuit 1610 a may be provided to the output analogneuromorphic circuit 1620 a that includes 15×1 resistive memories, thetwo hundred outputs generated by the smaller analog neuromorphic circuit1610 b may be provided to the output analog neuromorphic circuit 1620 bthat includes 15×1 resistive memories and so on.

The approximate sigmoid function may then be applied to each of therespective outputs generated by each of the output analog neuromorphiccircuits 1620(a-n). In doing so, the result of the approximate sigmoidfunction applied to each of the respective outputs generated by each ofthe output analog neuromorphic circuits 1620(a-n) may be similar to theresult generated when each of the 784 pixels of the initial image areapplied to the analog neuromorphic circuit that is of sufficient size tohandle each of the 784 pixels in a single analog neuromorphic circuit.The respective outputs generated by the output analog neuromorphiccircuits 1620(a-n) may be similar to that of the outputs generated by asingle analog neuromorphic circuit in that the respective outputs may berelied upon as being adequately correct as compared to the outputsgenerated by the single analog neuromorphic circuit.

Although the above example depicts smaller analog neuromorphic circuitsdepicted with 200×200 resistive memories as well as output analogneuromorphic circuits with 15×1 resistive memories where the initialanalog neuromorphic circuit is broken down vertically into the smalleranalog neuromorphic circuits, any quantity and any size of smalleranalog neuromorphic circuits and output analog neuromorphic circuits maybe implemented in breaking down the initial neuromorphic circuitvertically that is apparent to those skilled in the art.

It is to be appreciated that the Detailed Description section, and notthe Abstract section, is intended to be used to interpret the claims.The Abstract section can set forth one or more, but not all exemplaryembodiments, of the present disclosure, and thus, is not intended tolimit the present disclosure and the appended claims in any way.

While the present invention has been illustrated by the description ofone or more embodiments thereof, and while the embodiments have beendescribed in considerable detail, they are not intended to restrict orin any way limit the scope of the appended claims to such detail.Additional advantages and modifications will readily appear to thoseskilled in art. The invention in its broader aspects is therefore notlimited to the specific details, representative apparatus and method andillustrative examples shown and described. Accordingly, departures maybe made from such details without departing from the scope of thegeneral inventive concept.

What is claimed is:
 1. An analog neuromorphic system that implements aplurality of resistive memories, comprising: a plurality of inputvoltages applied to an analog neuromorphic circuit, wherein each inputvoltage represents a non-binary value that is incorporated into adot-product operation; a plurality of resistive memories with eachresistive memory configured to provide a resistance value to eachcorresponding input voltage; a plurality of output voltage signals thatis generated from the applying of each input voltage that represents acorresponding non-binary value to each corresponding resistance value toexecute the dot-product operation, wherein each output voltage signal isa binary value; and a plurality of comparator configurations that isconfigured to apply a non- linear smooth function to each correspondingoutput voltage signal to convert each output voltage signal from thecorresponding binary value to a non-binary value that is generated bythe dot-product operation.
 2. The analog neuromorphic system of claim 1,wherein each input voltage represents the non-binary value that isincorporated into the dot-product operation with a plurality of matrixvalues included in a matrix.
 3. The analog neuromorphic system of claim1, wherein the plurality of output voltage signals corresponds to eachcolumn of resistive memories of the analog neuromorphic circuit.
 4. Theanalog neuromorphic system of claim 3, wherein the plurality ofcomparator configurations is positioned at an output of each column ofresistive memories positioned in the analog neuromorphic circuit.
 5. Theanalog neuromorphic system of claim 4, wherein the non- linear smoothfunction is a sigmoid function that operates between a first value of 0and a second value of
 1. 6. The analog neuromorphic system of claim 5,wherein each comparator configuration is configured to: convert eachcorresponding output voltage signal to the corresponding non-binaryvalue as represented by each corresponding dot-product operation valueand each complemented dot-product operation value, wherein the dot-product operation value is an output of the dot-product operation for acorresponding column of resistive memories and the complemented dotproduct operation value is a complement of the output of the dot-productoperation for the corresponding column of resistive memories.
 7. Theanalog neuromorphic system of claim 6, wherein the comparatorconfiguration further comprises: a first op-amp configuration and asecond op-amp configuration that are positioned at the output of eachcorresponding column of resistive memories and are configured to scalethe corresponding output voltage signal to a value on the linear smoothfunction between the first value of 0 and the second value of
 1. 8. Amethod of executing a dot-product operation with an analog neuromorphicsystem that implements a plurality of resistive memories, comprising:applying a plurality of input voltages to an analog neuromorphiccircuit, wherein each input voltage represents a non-binary value thatis incorporated into the dot-product operation; providing a resistancevalue to each corresponding input voltage from a corresponding resistivememory included in a plurality of resistive memories; generating aplurality of output voltages form the applying of each input voltagethat represents a corresponding non-binary value to each correspondingresistance value to execute the dot-product operation, wherein eachoutput voltage is a binary value; and applying non-linear smoothfunction to each corresponding output voltage signal to convert eachoutput voltage signal from the corresponding binary value to anon-binary value that is generated by the dot-product operation.
 9. Themethod of claim 8, wherein each input voltage represents the non-binaryvalue that is incorporated into the dot-product operation with aplurality of matrix values included in a matrix.
 10. The method of claim8, wherein the plurality of output voltage signals corresponds to eachcolumn of resistive memories of the analog neuromorphic circuit.
 11. Themethod of claim 8, wherein the applying comprises: applying thenon-linear smooth function to each corresponding output voltage signalvia a plurality of comparator configurations positioned at an output ofeach column of resistive memories positioned in the analog neuromorphiccircuit to convert each output voltage signal from the correspondingbinary value to the non-binary value that is generated by thedot-product operation.
 12. The method of claim 11, wherein thenon-linear smooth function is a sigmoid function that operates between afirst value of 0 and a second value of
 1. 13. The method of claim 12,wherein each comparator configuration is configured to: convert eachcorresponding output voltage signal to the corresponding non-binaryvalue as represented by each corresponding dot-product operation valueand each complemented dot-product operation value, wherein the dot-product operation value is an output of the dot-product operation for acorresponding column of resistive memories and the complemented dotproduct operation value is a complement of the output of the dot-productoperation for the corresponding column of resistive memories.
 14. Themethod of claim 13, wherein the comparator configuration furthercomprises: a first op-amp configuration and a second op-ampconfiguration that are positioned at the output of each correspondingcolumn of resistive memories and are configured to scale thecorresponding output voltage signal to a value on the linear smoothfunction between the first value of 0 and the second value of
 1. 15. Ananalog neuromorphic system that implements a plurality of resistivememories, comprising: an input voltage applied to an analog neuromorphiccircuit, wherein the input voltage represents a non-binary value that isincorporated into a dot-product operation; a plurality of resistivememories that includes a resistive memory that corresponds to the inputvoltage applied to the analog neuromorphic circuit and is configured toprovide a resistance value to the input voltage; an output voltagesignal that is generated from applying the input voltage that representsa corresponding non-binary value to the corresponding resistance valueto execute the dot-product operation, wherein the output voltage signalis a binary value; and a resistance adjuster configured to: determinewhether a conductance value associated with the corresponding resistivememory is within a conductance value threshold, and adjust theconductance value of the resistive memory to be within the conductancevalue threshold when the conductance value associated with thecorresponding resistive memory deviates beyond the conductance valuethreshold.
 16. The analog neuromorphic system of claim 15, wherein theresistance adjuster is further configured to: deactivate each inputvoltage from a plurality of input voltages applied to each correspondingresistive memory, wherein each input voltage from the plurality of inputvoltages that is deactivated is set to 0.0V; activate a single inputvoltage from the plurality of input voltages applied to a singlecorresponding resistive memory, wherein the single output voltage isapplied to a horizontal wire that the single corresponding resistivememory is positioned; determine whether a single conductance valueassociated with the single corresponding resistive memory is within theconductance value threshold based on a single output voltage signal thatis generated from the activation of the single input voltage that isapplied to the single corresponding resistive mem ory; and adjust thesingle conductance value of the single corresponding resistive memory tobe within the conductance value threshold when the conductance valueassociated with the single corresponding resistive memory deviatesbeyond the conductance value threshold.
 17. The analog neuromorphicsystem of claim 16, wherein the resistance adjuster is furtherconfigured to: determine whether the single conductance value associatedwith the single corresponding resistive memory is within a conductancevalue threshold, wherein the conductance value threshold includes aminimum conductance level that the single conductance value is to beabove to satisfy the conductance level threshold and a maximumconductance level that the and single conductance value is to be belowto satisfy the conductance level threshold; and adjust the singleconductance value of the resistive memory to increase the singleconductance value to be within the conductance value threshold when thesingle conductance value associated with the single correspondingresistive memory is below the minimum conductance level and decrease thesingle conductance value to be within the conductance value thresholdwhen the single conductance value associated with the singlecorresponding resistive memory is above the maximum conductance level.18. The analog neuromorphic circuit of claim 17, further comprising: afirst op-amp configuration configured to: receive the single outputvoltage signal that is generated from the activation of the single inputvoltage that is applied to the single corresponding resistive memory;and generate a complemented dot-product operation value that isgenerated from the single output voltage signal and is based on thesingle conductance value associated with the single correspondingresistive memory.
 19. The analog neuromorphic circuit of claim 18,wherein the resistance adjuster is further configured to: determinewhether the complemented dot-product operation value associated with thesingle corresponding resistive memory is within the conductance valuethreshold; and adjust the single conductance value of the resistivememory to increase the complemented dot-product operation value to bewithin the conductance value threshold when the complemented dot-productoperation value associated with the single corresponding resistivememory is below the minimum conductance level and decrease thecomplemented dot-product operation value to be within the conductancevalue threshold when the complemented dot-product operation valueassociated with the single corresponding resistive memory is above themaximum conductance level.
 20. The analog neuromorphic circuit, whereinthe resistance adjuster comprises: a first comparator configured toreceive the complemented dot-product operation value that is applied tothe first comparator and a maximum conductance voltage signal that isapplied to the first comparator, wherein the maximum conductance voltagesignal represents the maximum conductance level; a second comparatorconfigured to receive the complemented dot-product operation value thatis applied to the second comparator and a minimum conductance voltagesignal that is applied to the second comparator, wherein the minimumconductance voltage signal represents the minimum conductance level; anincrease resistance adjuster configured to generate an increasedresistance signal when the complemented dot-product operation value isless than the minimum conductance voltage signal to increase the singleconductance value of the single corresponding resistive memory until thesingle conductance value is increased above the minimum conductancelevel; a decrease resistance adjuster configured to generate a decreasedresistance signal when the complemented dot-product operation value isabove the maximum conductance voltage signal to decrease the signalconductance value of the single corresponding resistive memory until thesingle conductance value is decreased below the maximum conductancelevel; and a no-change resistance adjusted configured to generate ano-change resistance signal when the complemented dot-product operationvalue is above the minimum conductance voltage signal and below themaximum conductance voltage signal thereby having the single conductancevalue of the single corresponding resistive memory remains unchanged.